Method and system for improvement of relevance of search results

ABSTRACT

A system and method for improving the relevance of search results is disclosed. Voters who may be human searchers or guides may review search results or other review items associated with a search request or other reference item. A review may be activated based on a usage indicator(s) which may improve utilization of guides. A vote by a voter may be weighted based on a voting history associated with the voter and one or more reference voters who may be designated by the system. A voter may be presented with a group of items for review including simultaneously. A number of comparison voting sessions or elections may be used to determine a rating or ranking of a review item associated with a reference item.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is related to and claims the benefit of the followingU.S. Provisional and U.S. patent applications, each of which areincorporated herein by reference in its entirety: U.S. ProvisionalApplication Ser. No. 60/913,944 entitled “KNOWLEDGE VETTING SYSTEM ANDMETHOD FOR SEARCH RESULTS”, by Scott A. Jones et al., filed Apr. 25,2007, U.S. Provisional Patent Application Ser. No. 60/980,049 entitled,“METHOD AND SYSTEM FOR IMPROVEMENT OF RELEVANCE OF SEARCH RESULTS”, byScott A. Jones, et. al., filed Oct. 15, 2007, U.S. application Ser. No.11/336,928, entitled “A SCALABLE SEARCH SYSTEM USING HUMAN SEARCHERS”,by Scott A. Jones, filed Jan. 23, 2006, now U.S. Pat. No. 8,065,286,U.S. application Ser. No. 11/779,502, entitled “ANONYMOUS SEARCH SYSTEMUSING HUMAN SEARCHERS”, by Scott A. Jones, et al., filed Jul. 18, 2007,U.S. application Ser. No. 11/780,297, entitled “METHOD, SYSTEM, ANDCOMPUTER READABLE STORAGE FOR PODCASTING AND VIDEO TRAINING IN ANINFORMATION SEARCH SYSTEM”, by Scott A. Jones, et al., filed Jul. 19,2007, U.S. patent application Ser. No. 11/774,852, entitled “METHOD ANDSYSTEM FOR SHARING AND ACCESSING RESOURCES”, by Scott A. Jones, et al.,filed Jul. 9, 2007, now U.S. Pat. No. 7,792,967, U.S. patent applicationSer. No. 11/835,016, entitled “ELECTRONIC PREVIOUS SEARCH RESULTS LOG”,by Scott A. Jones, et al., filed Aug. 7, 2007, now U.S. Pat. No.8,024,308, U.S. patent application Ser. No. 11/834,911 entitled,“METHOD, SYSTEM, AND COMPUTER READABLE STORAGE FOR AFFILIATE GROUPSEARCHING”, by Scott A. Jones, et. al., filed Aug. 7, 2007, now U.S.Pat. No. 7,792,967, U.S. Provisional Application Ser. No. 60/979,565,entitled METHOD AND SYSTEM FOR CREATION OF USER/GUIDE PROFILE IN AHUMAN-AIDED SEARCH SYSTEM, Attorney Docket 1918.1030P, filed Oct. 12,2007, and U.S. Provisional Application Ser. No. 60/980,010 entitledMETHOD AND SYSTEM FOR MATCHING INFORMATION SEEKERS AND SEARCHERS,Attorney Docket 1918.1031P, filed Oct. 15, 2007, in the United StatesPatent and Trademark Office, the disclosures of which are incorporatedherein by reference.

BACKGROUND

1. Field of the Invention

The present invention is related to search engine technologies and morespecifically to human-assisted search engines. A method and system forimproving the relevance of search results is disclosed.

2. Description of the Related Art

In general search engines are keyword driven systems. Keywords aregenerated from a user request and matched to target documents,advertisements, etc. This practice is generally established by suchservices as Google® or Dogpile®. The use of keywords allows a searchengine to produce relevant documents by using various methods of rankingdocuments, etc. An algorithm is applied to the content of each documentand a ranking is assigned. When a query is entered by a user, the searchsystem analyzes the query to extract keywords contained in the query andpresents documents in an order related to the ranking.

In the case of an automated search, the algorithm used to determinerelevance may be generalized in order to search various types ofdocuments. The algorithm used is designed to approximate human judgment.Such systems have enjoyed commercial success.

However these systems have weaknesses. For example, a more specificquery may not produce a more specific result. The number of keywords ina query will typically increase with the length of the query, and therelative strength of any keyword may decrease. Relevance of results maybe affected by factors such as search engine optimization. If analgorithm for determining relevance is known, a website designer mayoptimize his or her website to improve the website ranking. Relevance isnot a static property of a resource. Relevance may be affected by timeand/or other factors which can not be deduced from keywords. A typicalindexing strategy may be limited by the dynamic nature of sites, such asgame playing sites, auction sites, etc. Such resources may be easilyaccessible to a person, but are not easily indexed by a search engine.

In the case of a human-assisted search engine, human judgment may beused to determine the relevance of a result or resource. This techniqueis used by open directory projects such as DMOZ. Such systems are mainlyconstrained by the need to have a large number of human editors. Forexample, DMOZ had 75,000 editors, 500,000 categories, and 4,800,000websites in its index.

In light of the above and other existing problems, a scalable method andsystem for producing relevant targeted search results responsive to auser request which is based on use of human judgment and intelligencewould be greatly appreciated.

SUMMARY

A method and system are disclosed to improve the relevance of searchresults, search resources, advertisements, or other information to bepresented to a user responsive to a search request. A method isdisclosed for ranking items. Human judgment is used to rank the itemswith respect to a reference item(s). The order in which items such assearch results are presented to a user may be affected by the ranking. Asearch result(s) ranked higher than algorithmically produced results maybe displayed responsive to a search request(s).

Humans are used as voters. Guides or searchers may be presented withinformation of a reference item(s) and a review item(s) and may vote onthe relevance of an item(s). Users or information seekers may be used asvoters in at least one embodiment.

The method and system may be used to rank various types of information.For example, a response to a search request which is associated with acategory and a location may be reviewed by voters who are associatedwith a profile which may be associated with a search request. Anycombination of items might be used as a reference item(s) and/or areview item(s). For example, a category might be presented as areference item, and a topic such as a person's name might be a reviewitem, and voters might be selected based on an affiliation such asmembership in a group.

In at least one embodiment, the votes cast by voters are weighted. Forexample, one group of voters might have a vote weight of three, whileothers might have a vote weight of one, and others might have a voteweight of zero. The weight of a voter's vote may be determined based ona voting history associated with the voter. A voter's vote weight may beassigned based on his voting pattern relative to one or more referencevoters.

Various methods may be utilized to determine a ranking of an item basedon voting. If an item is referenced frequently, the item may be reviewedmore often than an item which is referenced infrequently. A reviewitem(s) associated with a reference item may be identified to be rankedbased on a number of times that a reference item has been used orpresented, or on a number of review items which have been associatedwith reference item, or based on various other criteria.

Additional aspects and/or advantages will be set forth in part in thedescription which follows and, in part, will be apparent from thedescription, or may be learned by practice of the invention.

These together with other aspects and advantages which will besubsequently apparent, reside in the details of construction andoperation as more fully hereinafter described, reference being had tothe accompanying drawings forming a part hereof, wherein like numeralsrefer to like parts throughout.

BRIEF DESCRIPTION OF THE DRAWINGS

Aspects and advantages of the disclosure will become apparent and morereadily appreciated from the following description of the embodiments,taken in conjunction with the accompanying drawings, of which:

FIG. 1 is a block diagram of an exemplary system embodiment.

FIG. 2 is a flowchart of a process of handling a search request.

FIG. 3 is a flowchart of a process of ranking items.

FIG. 4 is a flowchart of a process of determining a voter rating.

FIG. 4A is a diagram illustrating a process of determining voteweighting.

FIG. 5 is a diagram illustrating a process of ranking of items.

FIG. 6 is a flowchart of a method of reviewing items.

FIGS. 7A thru 7E are diagrams illustrating a process for pairing ofitems.

FIG. 8 illustrates a database record for a request.

FIG. 9 illustrates a database record for a guide.

FIG. 10 illustrates a database record for a user.

FIG. 11 illustrates a database record for a profile.

FIG. 12 illustrates a database record for a category.

FIG. 13 illustrates a database record for a result.

FIG. 14 illustrates a database record for a resource.

FIG. 15 illustrates a database record for an advertisement.

FIG. 16 is a flowchart of a process of creating a notification table.

FIG. 16A illustrates a database relationship.

FIG. 17 is a flowchart of a process of reviewing items.

FIG. 17A is a diagram illustrating a process of notification ofreviewers.

FIGS. 18A-18G illustrate GUIs.

FIG. 19 illustrates a GUI.

FIG. 20A-20C illustrate GUIs.

FIG. 21 illustrates a GUI.

FIG. 22 illustrates a GUI.

FIG. 23 illustrates a GUI.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Reference will now be made in detail to the present embodimentsdiscussed herein, examples of which are illustrated in the accompanyingdrawings, wherein like reference numerals refer to the like elementsthroughout. The embodiments are described below to explain the disclosedsystem and method by referring to the figures. It will nevertheless beunderstood that no limitation of the scope is thereby intended, suchalterations and further modifications in the illustrated device, andsuch further applications of the principles as illustrated therein beingcontemplated as would normally occur to one skilled in the art to whichthe embodiments relate.

A system and method is provided which allows a user or InfoSeeker™ whomay be any person to submit a search request or search query or requestto a search service and receive a response to the search request. Asearch system which allows communication to occur between a user(s), asearcher(s) or guide(s), a resource(s), and/or the search system allowsqueries to be processed. A database comprising information regarding auser(s), a guide(s), a resource(s), an advertisement(s), acategory(ies), a keyword(s), a tag(s), etc. is provided.

A search request or query submitted by a user, which may include akeyword(s), a category(ies), a fully formed question(s), a statement(s),any type of media, implicit data such as information associated with auser identifier(s) and/or device(s), which may request a response, mayreceive a response which is produced automatically and/or utilizing theassistance of one or more human searchers or guides. In at least oneembodiment a search request submitted by a user is compared to adatabase of search requests in order to determine if a matching searchrequest is found in the database. If a matching request is found, asearch result(s) associated with the matching search request(s) may bepresented to a user(s) responsive to a search request(s).

If multiple search result(s) are associated with a search request(s) thesearch result(s) may be ranked to determine an order in which the searchresult(s) will be presented to a user(s). A ranking of a searchresult(s) may be based on voting by voters. A search request isidentified, and other information such as a category(ies), geographicinformation and/or a profile(s) may be selected to be associated withthe search request. The search request is in this case a ‘referenceitem’ which is presented to voters. An indicator of items such as acategory(ies), geographic information, time information, or otherinformation associated with a reference item may also be presented tovoters. Voters may be selected based on association of a voter with anyor all information associated with a reference item. Voters may beprovided with the ability to record a preference regarding a review itemsuch as a search result(s). A vote(s) cast by a voter(s) may beweighted, (i.e. one voter may not get one vote). A group of voters maybe designated by system administrator(s), and members of the group mayhave specific voting privileges. Voter(s) may also be identified ashaving certain voting privileges based on experience, knowledge, and/orany other factor indicating ability to assist in providing relevantresult(s) pertaining to a request. Such voters may be referred to asPermanent Trusted Guides (PTG).

A vote cast by a voter(s) may be compared to a vote(s) cast by a PTG,and a rating of the voter may be calculated based on the comparison. Forexample, if a voter votes in agreement with a PTG, the voter's ratingmay increase, while if a voter votes contrary to the vote of a PTG, thevoter's rating may decrease. A PTG designation and/or a voter rating(s)may be associated with various items. For example, a guide who is a PTGwhen associated with a category such as “Sports>Football” might not be aPTG when associated with a category such as “Science>Astronomy”.Likewise a rating of a voter(s) may be associated with a category(ies),a profile(s), a location(s), a topic(s), a keyword(s), etc. The weightof a vote cast by a voter may be determined based on a rating associatedwith the voter.

In at least one embodiment, review items are voted upon and rated basedon votes cast by voters viewing a reference item and a number of itemsassociated with a reference item. For example, a voter(s) may bepresented with a category and a resource(s) which are to be reviewed, ora voter(s) may be presented with a search query, a category, and alocation, and an answer(s) which are to be reviewed. In at least oneembodiment, a reviewer may be selected based on items associated with areference item, which may not be displayed and/or selected by thereviewer. For example, a search query may be associated with a location,a profile, and one or more categories, and reviewers may be selected toreview search results associated with the query based on a rankingdetermined at least in part by ratings and/or rankings associated withthe location, profile and category(ies), but the profile, location,and/or category(ies) may not be presented to the reviewer(s). In atleast one embodiment, a reviewer may be allowed to select a keyword(s),category(ies), topic(s) and/or other information, which may be used toselect reference items and review items to be presented to the reviewer.

Reference and review items may be selected to be presented to a reviewerbased at least in part on criteria such as time, usage, agreements withadvertisers, a number of available and/or eligible reviewers, etc. Forexample, a reviewer may be presented with items in an order based on atime at which a review item was associated with a reference item, suchas presenting a review item which has been pending for the longest timefirst, or if multiple pairs of search requests and search results arepending review, the pairs which have been pending for the longest timeperiod may be preferentially presented to selected reviewers.

Review items presented for voting may be selected and presented usingvarious processes and GUIs. For example, a side-by-side comparison oftwo or more review items may be presented to a reviewer or voter. Avoter may be allowed to select a limited number of review items from alist of review items. For example, if two or more items are provided forreview, a voter may be allowed to vote for only one item, whileimplicitly voting against the others. Review items may be presented in alist, and a voter may be allowed to vote using a binary choice, or avoter may be allowed to forced-rank a list of review items from best toworst. Combinations of voting processes may be used.

A ranking of review item(s) may be determined using various processes.For example, a single review item may be paired with a number of otherreview items and a ranking may be determined based on the outcome of acomparison vote(s) between each pair of items. Review items might begrouped according to a current ranking and ranked again based on theoutcome of voting sessions regarding the grouped review items. Reviewitems might be ranked based on voting, and subsequently a new reviewitem(s) may be ranked using a comparison voting between the new reviewitem(s) and existing review items.

Ranking or rating of review items may be triggered by criteriaassociated with a reference item. For example, if a search result hasbeen presented to a user responsive to a search query more than a numberof times, a search result may be subjected to review by one or morevoters to determine whether the search result is still relevant.Alternately, if a search request is received more than a number oftimes, a ranking process for search results associated with the searchrequest may be initiated. In at least one embodiment, if a number ofsearch requests are associated with a category, tag, keyword, profileand/or location a ranking of review items which are associated with thecategory, tag, keyword, profile and/or location may be triggered.

The terms voice and speech are used interchangeably herein. A user, aresource, and/or a guide may establish a communication session using avoice service, a messaging service such as Short Messaging Service(SMS), Enhanced Messaging Service (EMS), Multi-media Messaging Service(MMS), Instant Messaging (IM), email, an internet portal or web page,regular mail or any other type of communication. A connection may beestablished using any device which is capable of utilizing acommunication service. For example, a wireless device such as a cellphone, PDA, smart phone, etc. might be used to establish a communicationsession using voice, SMS, IM, email and/or internet protocols. Adesktop, laptop or server system might be used to establish acommunication session. A landline phone, a specialized communicationterminal, or any other communication device might be used to establish acommunication session.

Communication between a guide, a user, a resource and/or a search systemmay include conversion of text to speech and speech to text. Any type ofmedia which can be sent or received using a communication system may bepart of a communication session. A communication session may beconducted using any or all communication service(s) associated with auser, a resource and/or a guide.

An advertisement may be transmitted including during any or allcommunication sessions between a user, a guide and/or a search system. Aresource, a guide, and/or an advertisement may be rated. Ratinginformation may be obtained from a user, a guide(s), and/or a searchsystem. Rating information may be used to select a resource(s), aguide(s), an advertisement(s) and/or any item(s) based on informationassociated with an item indicated in a database. The search service maybe compensated by advertising revenue.

As illustrated in FIG. 1, system 100 includes guide system(s) 105, 110,a network 115 such as the Internet, a search system 130, user system(s)or information seeker system(s) 135, 140, a database 120, which maycomprise various records, resource 155, and resource systems 145, 150.

While only a limited number of systems associated with a guide (alsoreferred to as a human searcher), resource (also referred to as a searchresource), user (also referred to as an information seeker or requester)and as a search system are depicted in FIG. 1, it is within the scope ofthe disclosure for multiple systems for guide, resource, informationseeker and search systems to be utilized.

Any user system (e.g., the user systems 135, 140) can be operated by aninformation seeker, who may be a person, to submit a search request tothe search system 130 and/or receive a search result(s) and/or otherinformation. Any guide system (e.g., the guide systems 105, 110) can beoperated by a human searcher to obtain a search result(s) for aninformation seeker located at a user system (e.g., the user systems 135,140). Any resource system (e.g., the resource systems 145, 150) may beoperated by a human provider of information and/or may be an automatedsystem which may provide a search result(s) and/or other information toa guide and/or a user, such as a search engine, a database, a localinformation source of a guide system such as a disk or removable memory,etc. A resource may not be accessible using the network 115. Forexample, a resource such as the resource 155 may be accessible to aguide operating a guide system such as the guide system 105. A resourcemight include printed materials, images, video, and/or audioinformation, a software application(s), any information accessible to aguide(s), a database(s), and/or any combination thereof.

The network 115 (FIG. 1) may be a global public network of networks (theInternet) and/or consist in whole or in part of one or more privatenetworks and communicatively couples the guide systems, the resourcesystems and the user systems with the other components of the systemsuch as the search system 130, and the database 120.

The search system 130 allows interaction to occur among the guidesystems 105, 110, the resource systems 145, 150 and the user systems135, 140. For example, an information search query(ies) can betransmitted from the user systems 135, 140 to the search system 130,where a search query(ies) can be accessed by the guide systems 105, 110and/or the resource systems 145, 150. Similarly, a search result(s)produced from the resource systems 145, 150 using the guide systems 105,110 in response to a search query(ies) submitted by the user systems135, 140 may be transmitted to the search system 130, where it may bestored by the search system 130 and/or may be transmitted to the usersystems 135, 140.

The search system 130 is communicatively coupled with the database 120.As will be described herein in further detail below, the database 120includes data that is processed in association with operation(s) of theembodiments. Although FIG. 1 illustrates the database 120 as a separatecomponent of the system, the database 120 may be integrated with thesearch system 130. Further, the records maintained in the database 120may be stored in any typical manner, including in a Network AttachedStorage (NAS), a Storage Area Network (SAN), etc. using any typical orproprietary database software such as DB2®, Informix®, Microsoft®SQLServer™, MySQL®, Oracle®, etc., and may also be a distributeddatabase on more than one server. Elements of the database 120 mayreside in any suitable elements of the system 100.

The user systems 135,140, the guide systems 105,110, the search system130 and the resource systems 145, 150 may include equipment, software,systems and personnel required to send and/or receive messages between auser system, a guide system, a resource system and/or the search systemusing the network 115. The database 120 includes information which mayallow the search system 130 to establish communication between the otherelements of the system.

A user system, a guide system, and/or a resource system may be a desktopor laptop PC or Mac®, a mobile phone, a smart phone, a PDA, a serversystem, a landline phone, a specialized communication terminal, aterminal connected to a mainframe, or any other communication device.The search system 130 may include one or more servers, computers, etc.After being presented with the disclosure herein, one of ordinary skillin the relevant art will immediately realize that any viable computersystem(s) or communication device(s) known in the art may be used asuser systems, guide systems, resource systems, and/or to implement thesearch system 130.

A guide may be required to register with the search system 130. As partof a registration process, at least one communication method isassociated with a guide. In at least one embodiment, a guide mayregister with the search system 130 and establish a username andpassword which are associated with the guide. A guide may login to thesearch system 130 using a web browser functionality of guide system 105,110 in order to communicate with the search system 130. Multiplecommunication services may be associated with a guide and may allow acommunication session to be established between a guide system such asthe guide system 105 and a user system, a resource system and/or thesearch system 130. Multiple identifiers of a guide may be associatedwith each other. Information such as IM credential(s), an emailaddress(es), a phone number(s), a URL, a username, etc. of a guide maybe identified which may allow the search system 130 to establish acommunication session between a guide system and a user system, aresource system, and/or the search system 130.

When a guide registers with the search system 130 the guide may beassociated with one or more keywords, categories, and/or otherinformation. For example a keyword(s) or category(ies) may be selectedby a guide, or may be associated with a guide based on a test(s)administered to a guide and/or other information provided during and/orafter a registration process. Information associated with a guide may bestored in the database 120 and may be used for purposes such as matchinga guide to a user request, determining and/or providing compensation fora guide, communicating with a guide, etc. as will be described furtherherein below.

A user may be identified by the search system 130. When a user systemsuch as the user system 135 establishes a communication session with thesearch system 130, an identifier of a user system is determined. Anidentifier of a user system may be associated with other informationregarding a user. A user system may be identified using an emailaddress, a telephone number, an IM credential, a username, or any otheridentifier which may be used to associate information with a user.Multiple identifiers of a user may be associated with each other. Usinginformation of communication services associated with a user, acommunication session may be established between a user system such asthe user system 135 and a guide system, a resource system and/or thesearch system 130. Information such as a keyword(s), a category(ies), auser profile(s), a previous search request(s), a search result(s), etc.may be associated with a user. Information of a user may be stored inthe database 120.

A resource, which may be a person(s), an entity(ies), a search engine, adatabase, a software application, a corpus of one or more types of mediasuch as text or printed information, images, audio, video, etc. or acombination thereof, may be identified by the search system 130.Information of at least one method of communication is associated with aresource system which allows a communication session to be establishedbetween the search system 130, a user system 135, 140, and/or a guidesystem 105, 110 and a resource system such as the resource systems 145,150. An identifier of a resource system may be associated with otherinformation regarding a resource. A resource system may be identifiedusing an email address, a telephone number, an IM credential, a resourceusername, a URL or other persistent identifier which may be used toassociate information with a resource. Multiple identifiers of aresource may be associated with each other. Using the information ofcommunication services associated with a resource, a communicationsession may be established between a resource system such as theresource system 145 and a user system, a guide system, and/or the searchsystem 130. Information such as a keyword(s), a category(ies), aprofile(s), or other information may be associated with a resource.Information of a resource may be stored in the database 120.

A resource such as the resource 155 and/or resources accessible via theresource systems 145, 150 may include any system(s), software, hardware,personnel and/or other facility(ies) which may provide information to aguide(s), a user(s), and/or the search system 130. For example, aresource may be a search engine, a database system, a library, apersonal hard drive and/or other local storage, printed materials,recordings of any sort, a software program(s), a person or person(s), anorganization, etc. A resource may be freely accessible to any user(s)and/or guide(s) and/or may be available on a restricted basis. Theresource system(s) 145, 150 may include resource(s) which are availableon an unrestricted and/or restricted basis. A resource may not beaccessible using the network 115, but may be accessible to selectedguide(s). For example, a resource such as the resource 155 may beaccessible to one or more guide(s) operating a guide system(s) such asthe guide system 105 using any type of communication. For example, aguide(s) may obtain information of an event(s) to provide a searchresult(s). Information in any form, such as printed media, audio and/orvisual information, software, hardware, etc. which may be accessible toa guide(s), a user(s) and/or an operator(s) of a resource system(s) maybe a resource.

The search system 130 may establish a communication session between anyuser system(s), guide system(s), or resource system(s) using informationindicated in the database 120. For example, the user system 135 mayestablish a voice communication session with the search system 130, thesearch system 130 may establish a voice communication session betweenthe user system 135 and the guide system 105, and the search system 130may establish a voice communication session between the user system 135and the resource system 145. While a voice communication session is usedin this example, any type of communication session using one or moreservices such as SMS, EMS, MMS, email, IM, chat, web basedcommunication, etc. may be established between any user system(s), guidesystem(s), and/or resource system(s) and/or the search system 130.

Information associated with a user(s), a guide(s) and/or a resource(s)may be obtained in various ways. For example, a registration process maybe performed using a web form(s) provided by the search system 130,and/or information may be obtained from an external database, and/orinformation may be obtained based on analysis of information indicatedby a user, a guide, and/or a resource. A ‘profile’ is one or morecharacteristics which may be associated with one or more individuals. Aprofile may include geographic data such as a street address, latitudeand longitude, etc., may include demographic information such as age,gender, race, income, family size, political affiliations, etc., mayinclude personality information such as results of psychometric testing,subjective evaluations of an individual, etc., may include affiliationinformation such as employment, club, activity, societal membershipinformation, information of a device(s), service(s), transaction(s) orany other information which might be associated with a user(s) and/or aguide(s).

As illustrated in FIG. 2, a process 200 for performing an informationsearch is provided.

In operation 202 (FIG. 2), a determination is made as to whether a useris logged into the search system 130 (FIG. 1). If it is determined inoperation 202 that a user is not logged-in control remains at operation202 and process 200 continues to wait. If it is determined in operation202 that a user is logged-in, control is passed to operation 205 andprocess 200 continues.

The determination in operation 202 may be made based on variouscriteria. A user may be determined to be logged into a search systembased on presence information associated with any or all communicationservices and/or devices of the user. For example, a user may bedetermined to be logged-in when a device associated with a user isaccessible using a communication service(s) of a user.

In operation 205, any pending previous search result(s) is presented toa user, user rating information may be obtained, and the database 120(FIG. 1) is updated. An advertisement(s) and/or other information may bepresented to a user before, during and/or after any or all searchresults are presented to a user. A user may be presented withinformation in any format(s) which is suitable for the device(s)associated with a user. A result(s) and/or an advertisement(s) may bepresented using one or more devices. The same or different informationmay be presented to a user at different times, based on conditions suchas the device(s) accessible to the user and/or the search system 130.For example, a user may be presented with an SMS message relating to asearch result, and may subsequently be presented with a URL associatedwith a search result(s) when accessing the search system using a browserfunctionality of a user device. Control is passed to operation 207 andprocess 200 continues.

In operation 207 (FIG. 2), a determination is made as to whether asearch query or search request is received. If it is determined inoperation 207 that a search query is not received, control is passed tooperation 202 and process 200 continues. If it is determined inoperation 207 that a search query is received, control is passed tooperation 210 and process 200 continues.

In operation 210, a determination is made as to whether a search querymatches an existing query indicated in the database 120 (FIG. 1). If itis determined in operation 210 (FIG. 2) that a query exactly matches anexisting query, control is passed to operation 215 and process 200continues. If it is determined in operation 210 that a partial match isfound to an existing query control is passed to operation 225 andprocess 200 continues. If it is determined in operation 210 that nomatch is found to an existing query control is passed to operation 235and process 200 continues.

In operation 215, the user ID and/or other information associated with auser entering a query is associated with a query or search request. Aresult(s) may be presented to a user and the database 120 (FIG. 1) isupdated. An order of presentation of a result(s) may be determined basedon rating data associated with the search result(s). In at least oneembodiment, a ratings may be based on a profile(s) associated with auser submitting a query, a profile(s) associated with a searcher(s) whoproduced a result(s), a profile(s) associated with a searcher(s) and/ora user(s) who have rated a result(s) associated with a query. Processesfor associating a user profile, a searcher profile(s) and a query aredescribed further in the related application U.S. ProvisionalApplication Ser. No. 60/980,010 previously mentioned and incorporatedherein by reference. Control is passed to operation 220 and process 200continues.

In operation 220, user ratings of a search result(s) may be obtained.The database 120 (FIG. 1) is updated. Usage information of a searchresult(s), an advertisement(s), etc. are recorded. Control is passed tooperation 202 and process 200 continues.

In operation 225, result(s) of query(ies) determined to partially matcha query submitted by a user may be presented to the user and/or aguide(s) as an alternative. The process of presentation may include adrop-down list, auto-fill functionality, or any other suitable process.Further, the determination may be made based upon content of a userprofile associated with the user and/or profile(s) associated with asearcher(s) that produced a search result(s) associated with thequery(ies). Control is passed to operation 230 and process 200continues.

In operation 230, a determination is made as to whether a user and/or aguide has accepted at least one of the presented results of queriespresented as matching the submitted query. If in operation 230 it isdetermined that at least one of the presented queries is accepted,control is passed to operation 215 and process 200 continues. If inoperation 230 it is determined that at least one of the presentedqueries is not accepted, control is passed to operation 235 and process200 continues.

In operation 235, a query is clarified. A parsing operation may beperformed to extract a keyword(s) and/or category(ies) associated withthe query. In at least one embodiment the parsing and categorization maybe done by the search system 130 (FIG. 1). In at least one embodimentthe parsing and categorization may be performed by a human guide.Interaction between a guide and an information seeker may occur toclarify the intent of a query. In operation 235 (FIG. 2) a categorizedand/or rephrased query is preferably obtained and a category(ies) and/orkeyword(s) may be associated with a query. In a preferred embodiment, acategorization and/or a keyword(s) selected by a human guide is comparedto that algorithmically selected by the search system 130 to providefeedback to the algorithmic selection. Control is passed to operation240 and process 200 continues.

In operation 240, a determination is made as to whether a clarifiedquery matches an existing query in the database 120 (FIG. 1). If inoperation 240 it is determined that a query matches an existing query,control is passed to operation 215 and process 200 continues. If inoperation 240 it is determined that a query does not match an existingquery, control is passed to operation 245 and process 200 continues.

In operation 245, a query record may be created, a query ID may beassigned, a user list is updated, and a searcher is selected. Aneligible searcher(s) may be determined using various processes.Preferred processes are described in the related U.S. patent applicationSer. No. 11/336,928, U.S. application Ser. No. 11/779,502, and U.S.application Ser. No. 11/780,297, all of the above identifiedapplications previously mentioned are incorporated herein by reference.An eligible searcher(s) may be presented with resources and a graphicaluser interface (GUI) for producing search results such as thosedescribed in the related U.S. patent application Ser. No. 11/774,852previously mentioned. Control is passed to operation 250 and process 200continues.

In operation 250, a determination is made as to whether a searchresult(s) has been returned by a searcher(s). If it is determined that aresult has not been returned, process 200 continues to wait. If apredetermined wait period is exceeded or if a searcher(s) indicates thatthe searcher(s) is unable to obtain a result an alternate searcher maybe selected, an automated search result from a search engine such asGoogle® or Yahoo!® may be returned, etc. in order to obtain a searchresult(s). If in operation 250 it is determined that a search result(s)has been returned, the search system database 120 (FIG. 1) is updatedand control is passed to operation 255.

In operation 255, a determination is made as to whether a user islogged-in to the search system 130 (FIG. 1). If it is determined inoperation 255 that a user is not logged-in, control is passed tooperation 260 and process 200 continues. If it is determined inoperation 255 that a user is logged-in control is passed to operation265 and process 200 continues.

In operation 265, a search result(s) is transmitted to a user. Userrating information may be obtained, the database 120 (FIG. 1) isupdated, control is passed to operation 202 and process 200 continues.

In operation 260, a search result(s) associated with a query and auser(s) are posted or stored in the database 120 (FIG. 1) with anindicator that the search result(s) are pending transmission to auser(s). Control is passed to operation 202 and process 200 continues.

In at least one embodiment, an association between a query and a searchresult may be created without requiring an information seeker. Thisfacility may be used to obtain query/result associations from anexternal source which may be produced in various ways. A query may becreated and a search result(s) may be associated with the query by aninteractive session between one or more guides (also referred to assearchers or providers or PaidSearcher™), for example, as part of atraining session. A query may be associated with a search result(s) bysubmission of a query to an automated search engine, such as Google®,Ask®, or Yahoo!®. A query and associated search result(s) may bemodified by annotations produced by processes such as those described inthe related U.S. Provisional Application Ser. No. 60/913,944 previouslymentioned. A query may be associated with a search result(s) and/orother information by review of a search process record such as thatdescribed in U.S. patent application Ser. No. 11/835,016, previouslymentioned, the contents of which are incorporated herein by reference.

In at least one embodiment, an indexed resource, such as a dictionary orencyclopedia may be processed to associate a query and a searchresult(s) and/or a category(ies), profile(s), etc. which may becomprised within the database 120 (FIG. 1). In some embodiments a querymay be associated with a search result(s) by selecting a precise searchresult(s) which may be used to create another query and/or an equivalentform(s) of the query which may be associated with the query.

Any number of equivalent queries may be associated with a searchrequest, and a search result(s) which may be associated with the searchrequest. As a search query or search request may be associated withmultiple search result(s), it may be desirable to rank the result(s)associated with a query to determine an order in which to present aresult(s) to a user who has submitted a query. In at least oneembodiment, a rating is associated with a result, a query and aprofile(s). A profile may contain information regarding a guide(s)and/or a user(s) who may rate a result(s). Result(s) associated with aquery may be presented in a different order based on a profileassociated with a query. A method for rating of a search result(s) isdisclosed in the related U.S. Provisional Application Ser. No.60/913,944, previously mentioned the disclosure of which is incorporatedherein by reference.

As illustrated in FIG. 3, a process 300 for ranking search resultsand/or items associated with a query using a human searcher(s) orvoter(s) is provided.

In operation 305 a determination is made as to whether eligible searchresult(s) are associated with a query. If it is determined in operation305 that there are not eligible results associated with a query, controlremains at operation 305 and process 300 continues. If in operation 305it is determined that there are eligible results associated with aquery, control is passed to operation 310 and process 300 continues. Inat least one embodiment, a query may be submitted to at least oneautomated search system and one or more automated search result(s) maybe associated with a query.

The determination of eligibility of a search result(s) in operation 305may be made based on various criteria. The criteria may include thepresence of at least one “human-touched” result which was selectedand/or reviewed by a person(s), at least one automatically producedresult, at least one certified result, a result(s) which has an averagerating above a pre-set threshold, whether a result has been evaluatedpreviously, any combination of the above, etc. In at least oneembodiment, having at least one “human-touched” search result (i.e., asearch result selected by a human guide) which has not been evaluatedand three other search results are the criteria for the determination inoperation 305. The determination in operation 305 may be made based onany suitable criteria as will be further described herein below.

In operation 310, a determination is made as to whether a profile is tobe used as part of the process for rating results. If it is determinedin operation 310 that a profile is to be used, control is passed tooperation 315 and process 300 continues. If it is determined inoperation 310 that a profile is not to be used, control is passed tooperation 320 and process 300 continues.

The determination in operation 310 may be made based on variouscondition(s). For example, if a number of searchers associated with aprofile which is associated with a query is greater than a predeterminedvalue, a profile-based comparison may be conducted. Alternately, if anumber of users associated with a profile associated with a queryexceeds a predetermined value, profile-based comparison may beconducted. Moreover, other criteria may be used for the determination inoperation 310 without departing from the spirit and scope of theembodiments described herein.

In operation 315, a profile to be used for reviewing a result(s)associated with a query is selected. The selected profile may be anyprofile. If no ratings are associated with a result(s), a query and aprofile, a current rating(s) associated with a query and a result(s) maybe used to determine a rating(s) associated with a query, a result(s)and a profile. A profile may be selected based on information associatedwith a query or available to the search system 130 (FIG. 1). Forexample, if a number of user(s) submitting a query are associated with aprofile(s), the profile may be associated with the query for the purposeof ranking a search result(s). Likewise for a query which may require asubjective judgment, one or more profiles which may differentiate agroup(s) of voters may be associated with a search query. Control ispassed to operation 320 and process 300 continues.

In operation 320, the results are sorted for a comparison voting. Thesorting may be performed in various ways. For example, a voting processsuch as that described in related U.S. Provisional Application Ser. No.60/913,944 previously mentioned may be used to determine a ranking forany or all result(s). Alternately, results from various sources may beassigned a rating based on the source of the result(s). For example, aresult produced by an affiliate organization as described in relatedU.S. application Ser. No. 11/834,911, previously mentioned, may be givena high rating, as it was produced by a trusted source, and a result froman automated search resource might receive a medium rating, and a resultobtained from a search by a low-ranking guide might receive a lowrating. Results may be sorted from the highest rating to the lowestrating. Alternately, a random ordering might be selected. Control ispassed to operation 325 and process 300 continues.

In operation 325, results are grouped. Grouping may be based on theranking operation performed in operation 320. Results may be grouped ingroups of two or more. The grouping may be based on various processes.In one embodiment, the highest rated result is matched with the lowestrated result, the second-highest rated result is matched with the secondlowest rated result, etc. as illustrated in FIG. 7A. Alternately, thehighest rated result is matched with the second highest rated result,and the lowest rated result is matched with the second lowest ratedresult as illustrated in FIG. 7B. In a case where there is no parity(i.e., the total number of results is not divisible by the number ofresults in a group) one or more groups may have added members to allowany result(s) to be voted upon. Alternately, one or more results may beremoved from the comparison process. In at least one embodiment, agroup(s) consists of two results. In at least one embodiment, a humanproduced ‘nugget’ result is paired with each of three results asillustrated in FIG. 7C. Control is passed to operation 330 and process300 continues.

In operation 330, comparative voting is conducted on members of one ormore groups from operation 325. In at least one embodiment, a voter isallowed to select only one member of a group as a preferred result. Avoter(s) may be presented with one or more result groups, along with aquery which is associated with a group(s) of results. If profile basedvoting is selected, voters may be eligible to vote based on a profileassociated with a voter and a selected profile(s) from operation 315.Control is passed to operation 335 and process 300 continues.

In operation 335, a determination is made as to whether comparisonvoting is complete. If in operation 335 it is determined that comparisonvoting is not complete, control is passed to operation 330 and process300 continues. If in operation 335 it is determined that comparisonvoting is complete, control is passed to operation 340 and process 300continues.

The determination in operation 335 may be made based on variouscriteria. For example, the determination may be based on whether anumber of votes associated with one or more result group(s) exceeds apredetermined value, whether a total number of votes for any resultgroup exceed a predetermined threshold, or any other criteria. In atleast one embodiment, an election (or comparison of a group of results)is determined to be complete when at least three eligible guides havevoted. In at least one embodiment, a comparison vote is determined to becomplete based on whether a member of a group of guides has voted.Voting may be determined to be complete when all groups created inoperation 325 have been determined to have met the completion criteriafor a voting process. For example, if there are 4 groups of resultsbeing compared, and each comparison is determined to be complete whenthe total number of votes for any member of a group exceeds a value(e.g., 5 votes), the voting may be determined to be complete when onemember of each group of results has received at least 6 votes.

In operation 340, the search results are sorted based on the comparativevoting sessions conducted in operation 330. A result(s) which receives ahigher number of votes may be ranked higher than a result which receivesa lower number of votes. For results which are in different groups aprocess for inter-group ranking may be applied. Various processes forinter-group ranking may be used. For example, a higher rated result(s)may be grouped with higher rated result(s), and a lower rated result(s)may be grouped with a lower rated result(s). Various exemplary processesfor grouping results after completion of a comparative vote for agroup(s) are illustrated in FIGS. 7D and 7E. A winning result(s) may begrouped with a winning result in a pre-seeded fashion as illustrated inFIG. 7D. A ‘best’ result (also referred to herein as a ‘nugget’) may bethe winner of the ‘tournament’ pairing. Alternately, results may begrouped in a ‘round-robin’ fashion as illustrated in FIG. 7E. A ‘best’result may be one which has the highest won/lost percentage. In at leastone embodiment, a ‘nugget’ must win a comparative vote versus each ofthree other result(s), as illustrated in FIG. 7C. Control is passed tooperation 345 and process 300 continues.

In operation 345, a determination is made as to whether a resultscomparison operation is complete. If it is determined in operation 345that a comparison operation is complete, control is passed to operation305 and process 300 continues. If it is determined in operation 345 thata comparison operation is not complete, control is passed to operation325 and process 300 continues.

The determination of completion in operation 345 may be based oncriteria such as the completion of a predetermined number of comparativevoting sessions, no change in the ranking of the results, the expirationof a predetermined time period, etc. In at least one embodiment thedetermination in operation 345 (FIG. 3) is made based on completion ofthree elections or comparison votes relating the three pairingsdescribed in FIG. 7C (i.e., a human-touched result versus three othersearch results).

Exemplary processes for comparative result grouping are illustrated inFIGS. 7D and 7E. Other grouping processes and other end conditions mightbe utilized to produce a ranking of a predetermined number of resultsusing a predetermined number of voting comparisons.

The ranking of results associated with a query may be used to determinean order in which results are presented to a user when a query matchingthe query associated the search result(s) is submitted. The top-rankedresult may preferably be presented to a user first. Results may bepresented to a user in rank order and/or in an order which may bedetermined based on data indicated in the search database 120 (FIG. 1).

As illustrated in FIG. 4, a process 400 for determining a vote weightfor human searcher(s) or voter(s) is provided.

In operation 405, a determination is made as to whether voting resultsare available. If it is determined in operation 405 that voting resultsare not available, control remains at operation 405 and process 400continues. If in operation 405 it is determined that voting results areavailable, control is passed to operation 410 and process 400 continues.In at least one embodiment, completion of a voting session is used todetermine if voting results are available.

In operation 410, the votes associated with a voter or reviewer and aquery or other reference item(s) and a review item such as a searchresult(s) are compared to the votes of one or more reference voters. Areference voter(s) may be a group of one or more voters identified bythe search system 130 (FIG. 1). For example, one or more guides may bedesignated as a system representative or Permanent Trusted Guide (PTG).The designation of PTG may be associated with various types ofinformation. For example, a guide may be designated as a PTG whenassociated with one or more categories, or regions, or topics, orprofiles. A voter may be a reference voter based on historical votingpatterns. For example, if a guide has voted in alignment with a PTG apercentage of the time, and/or an aggregate number of times, the guidemay be designated as a Trusted Guide (TG). A group of reference guidesmay be identified based on testing activities, qualifications, or otherinformation indicated in the database 120 (FIG. 1). Control is passed tooperation 415 and process 400 continues.

In operation 415, the votes associated with a voter or reviewer and aquery or other reference item(s) and a review item such as a searchresult(s) are compared to votes of the majority or plurality of votersassociated with the review item and the reference item(s). Suchcomparisons may be used to determine a vote weighting associated with avoter. In at least one embodiment, the vote of a PTG and/or a TG may becompared to the vote of a majority or plurality of voters, which mayaffect a vote weight associated with a voter(s). Control is passed tooperation 420 and process 400 continues.

In operation 420, a voter rating is calculated for a voter(s). Forexample, a rating function such as the voter rating function 455depicted in FIG. 4A may be computed. The total number of votes contraryto the vote of a PTG(s) may be subtracted from the total number of votesin agreement with a PTG(s) in order to compute a rating of a voter.Alternately the ratio of votes in agreement with a TG and/or PTG tovotes contrary to a PTG and/or TG may be utilized to calculate a ratingof a voter. A rating function for a voter may be based on anymathematical relationship between votes cast by a voter and votes castby a reference voter(s). In at least one embodiment, a differential ofaligned versus misaligned votes is applied to an exponential function todetermine a rating of a voter. Similarly a rating function may includethe alignment between a voter and a majority and/or plurality of voters.Control is passed to operation 425 and process 400 continues.

In operation 425, a vote weight is calculated for a voter(s). Forexample, a vote weighting function such as the weighting function 485(FIG. 4A) is evaluated. The weight assigned to a voter is based on thevalue of the rating function, as is further described herein below. Anyweighting function may be utilized to determine a vote weight associatedwith a voter. For example, a PTG may have a vote weight which is greaterthan that of other voter(s), and greater than that associated with thevote weighting function. Likewise, a TG may have a higher vote weightthan a guide who has a limited voting history. Likewise, a guide whovotes contrary to a PTG and/or TG may receive a low vote weight. In atleast one embodiment, a TG has a vote weight of 2, a guide initially hasa vote weight of 1, and a guide who consistently votes contrary to a PTGand/or TG has a vote weight of 0. Control is passed to operation 430 andprocess 400 continues.

In operation 430, the search database 120 (FIG. 1) is updated.Information of a rating and/or vote weight associated with a voter(s)and/or a profile(s), a category(ies), keyword(s), etc. is recorded.Control is passed to operation 405 and process 400 continues.

FIG. 4A illustrates an exemplary relationship 450 between votes cast bya voter, a rating of a voter, and a vote weight assigned to a voter.

A number or percentage of aligned and/or contrary votes by a voter isrepresented on the x-axis in FIG. 4A. A voter rating function is appliedto the computed number of aligned and/or misaligned votes as representedby the triangle indication curve 455. A threshold may be applied to thevote rating function 455 as represented by the threshold values 460 and465 represented by the dashed lines in FIG. 4A. The threshold valuesrelate the voter rating function to the vote weighting functionillustrated by the diamond indication curve 485. Associated with eachthreshold value, is a vote weight. For example, when the rating functionexceeds the high threshold 460, a high vote weight designated by thehigh weight value 470 is assigned to a voter. Likewise when the ratingfunction is below the upper threshold 460 and above the lower threshold465, the vote weight is the middle weight value 475, and when the ratingfunction is below the lower threshold 465, the vote weight is the lowweight value 480. While two thresholds and three vote weights have beenused for the purposes of illustration in FIG. 4A, any number ofthreshold values and vote weight values might be utilized. Likewisevarious types of rating functions may be utilized which may combine oneor more elements of voting behaviors associated with a voter may beutilized within the spirit and scope of the disclosure herein.

FIG. 5 illustrates a process for ranking items based on comparisonsbetween pairs of items. A review item container 500 or list whichcontains two or more review items which are to be ranked is created.Each item is assigned an item identifier (ID) which may be stored in theitem ID list 505 of the review item container 500. A ranking is assignedto each item as it is added to the review item container 500. A rankingassociated with each item ID indicated in the item ID list field 505 isstored in the item rank list 510. In at least one embodiment, the itemID list field 505 and the item rank list field 510 are linked by forexample a pointer. Any number of items may be indicated in the reviewitem container 500. In at least one embodiment, the number of items inthe review item container 500 is limited to a value T as indicated inthe rank field 530 of the item rank list 510 of the review itemcontainer 500. The number of items to be presented to a user may bedetermined by a value P as indicated in the rank field 520 associatedwith the last item to be presented. An item to be added to the reviewitem container 500 is initially compared to an item at the position S asindicated in the rank field 525 associated with the initial comparisonitem.

When an item(s) is to be added to the review item container 500 a listof review items may be sorted. For example, a list of search results maybe sorted based on time stamp information associated with the searchresults. An initial item may be inserted into the review container 500at the location S. In the case of search results sorted by time stamp,the oldest search result might be added at the starting rank position S.A review item is then compared to the item at the position S. If the newitem wins the comparison (i.e. voting) the new item is inserted at theposition above the initial comparison item, if that location isunoccupied. If that location is occupied, the new item is compared tothe item at that location. If the new item wins the comparison, it isthen promoted to the next higher rank and compared to a result at thatposition, if any. The process continues until the new item eitherreaches the top of the list (i.e. the highest open ranking, or the firstposition), or loses a comparison. If the new item reaches the top of thelist, it is inserted at the top rank position and all other results aremoved downward, if needed. If the new item loses a comparison beforereaching the top of the list, it is inserted at the position below theitem which won the comparison, and the other results are shifted down byone position. If the new item loses the comparison to the item at thestart position S, the new item is then compared to the item, if any atthe next lower position in the Item ID list 505. If new item wins thecomparison, it is inserted at that location and the other items areshifted down by one position. If the new item loses, it is compared tothe next lower item until it wins a comparison, reaches the bottom ofthe list (i.e., an unoccupied position), or loses to the item at thelast tracked position T.

By changing a number of items tracked (T) relative to the startcomparison position (S) the number of elections or comparisons requiredto discard a review item may be modified. A smaller value of T relativeto S reduces the number of comparisons, and the associated cost toremove an item from the review container 500. Adjusting the number ofitems presented (P) relative to S changes the number of comparisonsrequired for a review item to be presented to a user. By decreasing Prelative to S, the number of comparisons and the associated cost for asuccessful item to be presented to a user is increased. By varying thevalues P, S and T, the cost of the review process may be modified.Different types of reference and review items may utilize differentvalues of P, S and T as determined by the operators of the system 100(FIG. 1).

As illustrated in FIG. 6, a process 600 for reviewing a review item(s)associated with a reference item(s) is provided.

In operation 605, a determination is made as to whether a reference itemhas been utilized. If it is determined in operation 605 that an item hasnot been utilized, control remains at operation 605 and process 600continues. If in operation 605 it is determined that an item has beenutilized, control is passed to operation 610 and process 600 continues.In at least one embodiment, presenting a search result(s) responsive toa search request is used to determine whether an item has been utilized.In at least one embodiment, access to a search resource (the reviewitem) by a searcher or guide responsive to a search request is used todetermine whether an item has been utilized. Usage of a referenceitem(s) and/or a review item(s) may be used to determine if an item hasbeen accessed. For example, if a search request which matches anexisting search request is submitted by a user, it may be determinedthat the reference item (i.e. the search request) has been utilized.

In operation 610, a determination is made as to whether a usage limithas been exceeded. If it is determined in operation 610 that a usagelimit has not been exceeded, control is passed to operation 605 andprocess 600 continues. If in operation 610 it is determined that a usagelimit (count) has been exceeded, control is passed to operation 612 andprocess 600 continues. A usage limit may be established in various ways.In at least one embodiment, a guide may determine a usage limit (count)for an item(s). A usage limit (count) for an item may be determinedbased on content of the item. For example, a search result which isfactual information such as historical data or the definition of a wordmay have a high usage limit as compared to news items or sportsinformation which may have a low usage limit.

In operation 612, a determination is made as to whether an abbreviatedreview is to be performed. If it is determined in operation 612 that anabbreviated review is not to be performed, control is passed tooperation 617 and process 600 continues. If in operation 612 it isdetermined that an abbreviated review is to be performed, control ispassed to operation 615 and process 600 continues. The determination inoperation 612 may be made based on various criteria. For example, thecontent of or data associated with a reference item may be utilized todetermine if an abbreviated review is to be performed. In at least oneembodiment, a number of available reviewers may be used to determine ifan abbreviated review is to be performed. An abbreviated review may beperformed based on pricing information associated with a referenceitem(s) and/or a review item(s). For example, a keyword(s) and/orcategory(ies) associated with a high bid amount may receive a differentlevel of effort than a keyword(s) and/or category(ies) associated with alower bid amount, or a query associated with a high payment per resultmight receive a different level of effort than a query associated with alow payment per result. Any information indicated in the database 120(FIG. 1) may be used to determine whether an abbreviated review is to beconducted. In at least one embodiment, an abbreviated review may not beperformed. An abbreviated review may be selected in order to modifyeffort associated with performing a review of a review item(s)associated with a reference item(s).

In operation 615, a reference item(s) and a review item are presented toa reviewer. For example, information of a search request and a searchresult associated with the search request may be presented to a guide. Areviewer may be selected based on information associated with thereviewer. For example, a guide associated with a keyword(s), acategory(ies) or a profile(s) associated with a search request may beselected to review a search result associated with the search request.Alternately, a guide associated with a category might be might beselected to review a search resource or an advertisement associated withthe category. Control is passed to operation 620 and process 600continues.

In operation 617, information of a reference item(s) is delivered to areviewer (expert). For example, information of a search request may bepresented to a guide. A reviewer may be selected based on informationassociated with the reviewer. For example, a guide(s) associated with akeyword(s), a category(ies) or a profile(s) associated with a searchrequest may be selected to receive information of the search request.Alternately, a guide(s) associated with a category might be selected toreceive information of the category. In at least one embodiment, areviewer may be an automated system and/or an API. Control is passed tooperation 622 and process 600 continues.

In operation 622 a new review item is obtained. For example, a selectedguide or researcher may perform an information search and provide asearch result responsive to a request presented in operation 617.Alternately, a guide(s) associated with a category might identify asearch resource, an advertisement, training materials, or any other typeof information responsive to the category information presented inoperation 617. In at least one embodiment, information of a referenceitem may be submitted to an automated search in order to obtain a newreview item. Control is passed to operation 627 and process 600continues.

In operation 627 a review is performed which includes an item(s)obtained in operation 622. In a preferred embodiment, a review processsuch as those described herein with respect to FIG. 4, FIG. 5, and FIG.7 may be used. For example, a selected guide(s) may review any or allsearch results associated with a search request. Likewise, a guide(s)associated with a category might review a search resource(s), anadvertisement(s), training material(s), or any other type(s) ofinformation associated with the category. Any review item(s) associatedwith a reference item may be reviewed. Control is passed to operation645 and process 600 continues.

In operation 620 a determination is made as to whether a reviewerrejects a review item(s). If it is determined in operation 620 that areviewer does not reject a review item(s), control is passed tooperation 650 and process 600 continues. If in operation 620 it isdetermined that a reviewer rejects a review item(s), control is passedto operation 625 and process 600 continues.

In operation 650 an item usage indicator is updated. Informationregarding the review of the item is recorded in the database 120 (FIG.1). For example, an identifier of the reviewer, time and dateinformation associated with the review, etc. may be recorded. Control ispassed to operation 605 and process 600 continues.

In operation 625 a new review item is obtained. For example, a searcheror guide may obtain a search result responsive to a search request whichwas presented to a reviewer in operation 615 or the search request maybe submitted to an automated search system. Likewise a new searchresource might be located by a guide, or a new advertisement might beobtained from a database of advertisements. Control is passed tooperation 630 and process 600 continues.

In operation 630 a comparison review by a reviewer is conducted. Forexample, a searcher or guide may be presented with a search request, thenew search result obtained in operation 625 and the previous searchresult which was rejected in operation 620. Control is passed tooperation 635 and process 600 continues.

In operation 635 a determination is made as to whether a reviewerselects the new review item. If it is determined in operation 635 that areviewer does not select the new review item, control is passed tooperation 650 and process 600 continues. If in operation 635 it isdetermined that a reviewer selects the new review item, control ispassed to operation 640 and process 600 continues.

In operation 640 the new review item is associated with the referenceitem(s). For example, a new search result may be associated with asearch request, or a new resource may be associated with a category,etc. Control is passed to operation 645 and process 600 continues.

In operation 645 the item usage counter is updated to reflect theresults of the comparison process. Information associated with referenceitems such as a search request, a keyword(s), a category(ies), aprofile(s), etc. and/or review items such as a search result(s), asearch resource(s), an advertisement(s), etc. may be recorded ormodified. Control is passed to operation 605 and process 600 continues.

An exemplary process for grouping review items is illustrated in FIG.7A. A list of items has length N. The highest rated item is paired withthe lowest rated item, the second highest with the second lowest, etc.Such a pairing process may be used in conjunction with the process 300illustrated in FIG. 3. Results of comparisons may be used to re-computea ranking of the items in the list. For example, after an initialcomparison, the winning items may be compared to each other. In such acase, N−1 comparisons may be used to determine a ‘best’ item.

An exemplary process for grouping review items is illustrated in FIG.7B. A list of items has length N. The top-ranked result is paired withthe second ranked result, and the lowest ranked result is paired withthe second lowest ranked result. Such a pairing process may be used inconjunction with the process 300 illustrated in FIG. 3. Results ofcomparisons may be used to re-compute a ranking of the items in thelist. For example, after an initial comparison, only the winning itemsmight be compared to winning items, and losing results may be comparedto each other.

An exemplary process for grouping of review items is illustrated in FIG.7C. A single item is compared to each of three other items. If the itemis a winner in each of the three comparisons, it may be designated as atop-ranked item which is to be used. For example, if only one searchresult is to be presented to a user(s) responsive to a search request, ahuman touched search result which is determined to be superior to threeautomated search results produced responsive to the search request maybe presented.

An exemplary process for determining a ranking of items is illustratedin FIG. 7D. A list of N items is grouped, using for example the groupingprocesses illustrated in FIG. 7A or FIG. 7B, and successive comparisonsare conducted between the members of the list. After each round ofvoting, the list of losing items may be sorted, and inserted at thelower end of a ranking based on their prior ranking. The best item isthe item which has not been determined to be inferior in any comparison.The number of comparisons is N−1 when groups of 2 items are used.

An exemplary process for determining a ranking of items is illustratedin FIG. 7E. A list of N items is grouped in pairs. Each of the N itemsis compared to each of the other items. The ranking of an item isdetermined based on the total number of comparisons in which it wasjudged to be superior. Every item is compared to every other item, andthus the ranking takes more comparisons (i.e., N factorial), but mayprovide a more consistent ranking of items as all items are comparedregardless of rank. The processes illustrated in FIGS. 7A through 7E areused for the purposes of illustration, and other methods for pairingitems for comparison and ranking will readily occur to one of ordinaryskill in the relevant art.

As illustrated in FIG. 8, an exemplary request record 800 is provided,of which one or more may be associated with or resident in the searchdatabase 120 (FIG. 1), and may include a request identification (ID)field 805, a request content field 810, a request user ID field 815, arequest guide ID field 820, a request category ID field 825, a requestprofile ID field 830, a request result ID field 835, a request resultrank field 840, a request advertisement ID field 845, a requestadvertisement rank field 850 and a request usage information field 855.

A request record may be created by various processes, including a guidedsearch session, an interactive training session, a non-interactivetraining session, may be imported from an external resource accessibleto the search system, submission of a search request to a resource(s)such as a search engine(s), etc.

The request ID field 805 preferably contains a unique identifier of arequest, which is preferably used consistently. For example, in at leastone embodiment, the request ID field 805 can include a randomlygenerated numerical code, and/or a text string indicating the content ofa query. A request ID serves to distinguish the request recordassociated with a request from a request record associated with otherrequests. Other unique identifiers of a search request may be utilizedwithout departing from the spirit and scope of the embodiments.

A unique identifier is assigned to a request when it is entered into thesearch database. If it is determined that a search request is identicalto a request in the search database, the request may be assigned thesame request ID and information of the search request may be added tothe request record associated with the request ID. A query or searchrequest may include information that is not explicitly entered by auser/requester. For example, location information, profile information,etc. may be associated with a query which might differentiate the queryfrom a query in the search database 120 (FIG. 1). A search request whichis determined to be unique may receive a new request ID and anassociated query record. As illustrated in FIG. 8, ‘Request1’ is therequest ID associated with the request record 800 (FIG. 8).

The request content field 810 includes information regarding the contentof a search request. For example, text associated with a query submittedby a user may be indicated in the request content field 810. Suchinformation may be processed in order to associate a keyword(s) and/orcategory(ies) or other information with a search request. In at leastone embodiment, a structured query which conforms to a specific contextmay be indicated in the request content field 810. User queries whichare associated with a structured query which may have been submitted bymore than one user may be indicated in the request content field 810.Information such as audio recordings, images, etc. which are associatedwith a search request may be indicated in the request content field 810.Using the example illustrated in FIG. 8, the query ‘What is the bestThat restaurant in Indy?’ is the query content associated with therequest record 800. This may indicate that the query ‘What is the bestThat restaurant in Indy?’ may be presented to a voter(s) selected tovote regarding a search result(s) or other item(s) associated with therequest ‘Request1’.

The request user ID field 815 includes information of a user(s) thatsubmitted a query to the search system 130 (FIG. 1). The request user IDfield 815 (FIG. 8) may be used to associate a user(s) with a query.Multiple users may submit an equivalent search request to the searchsystem 130 (FIG. 1). A search result(s) and/or other item(s) associatedwith a search request may be presented to a user(s) and/or a guide(s)responsive to a search request in an order based at least in part on aranking of the item(s) associated with the search request. Anidentifier(s) of a number of users may be indicated in the request userID field 815. Information indicated in the request user ID field 815 maybe used to obtain information of a user(s) using a record(s) such as theuser record 1000 illustrated in FIG. 10. Using the example illustratedin FIG. 8 the users ‘User1’ and ‘User2’ have been associated with therequest ‘Request1’

The request guide ID field 820 includes information of a guide(s) whoare associated with a search request. For example, if a guide obtains asearch result(s) responsive to the search request ‘Request1’, anidentifier of the guide may be indicated in the request guide ID field820. Likewise, if a guide was selected to respond to a search request,an identifier of the guide may be indicated in the request guide IDfield 820. Information indicated in the request guide ID field 820 maybe used to obtain information associated with a guide using a record(s)such as the guide record 900 illustrated in FIG. 9. Using the exampleillustrated in FIG. 8 the guides ‘Guide1’ and ‘Guide2’ have beenassociated with the request ‘Request1’.

The request category ID field 825 includes information of acategory(ies) and/or keyword(s) associated with a request. The contentof the request category ID field 825 may be modified by an automatedclassification of a request. A human may select a category(ies) and/orkeyword(s) which is associated with a request. For example, a requestmay be associated with the category “Science>Astronomy”, while anothermay be associated with the category “Arts>Music. A category may beassociated with a search resource(s) such as a search engine, or awebsite, etc., a guide(s), an advertisement(s), etc. An item(s)associated with a category may be presented to a guide(s) and/or auser(s) when a search request associated with a category is submitted tothe search system 130 (FIG. 1). An item(s) associated with a categoryand/or other information may be presented to a voter(s) for review.Association of a category with a request may be used to select itemssuch as an advertisement(s) associated with the category to be presentedto a user(s) responsive to a search request. Association of a categorywith a search request may be used to select a voter(s) associated withthe category to vote regarding an item associated with the searchrequest. Using the example illustrated in FIG. 8, the categories‘Category1’ and ‘Category2’ are associated with the request ‘Request1’.

The request profile ID field 830 includes information of a profile(s)which is associated with a search request. For example, a profile may beassociated with a search request based at least in part on a profile(s)associated with a user(s) associated with a search request. For example,a geographic profile which is common to one or more users associatedwith a search request may be associated with a search request, or aguide may determine that a profile is to be associated with a searchrequest based on the content of a search request(s) associated with auser(s). A guide selected to respond to a search request may be selectedat least in part based on a profile(s) associated with the searchrequest. A voter(s) may be selected to vote regarding an item(s)associated with a search request based at least in part on a profile(s)associated with a search request. Using the example illustrated in FIG.8, the profile ‘Profile1’ is associated with the request ‘Request1’.

The request result ID field 835 includes information of a result(s)which is associated with a search request. For example, a result may beassociated with a search request when a guide obtains the search resultresponsive to the search request. For example, if a search request issubmitted to the search system 130 (FIG. 1), a guide may be selected,and the guide may provide a search result(s) responsive to the request.Alternately, a search result(s) may be associated with a search requestif the search request is submitted to a resource such as a searchengine, or other resource system. Any number of search results may beassociated with a search request. Using the example illustrated in FIG.8, the result ‘Result1.1’, the result ‘Result1.2’, the result‘Result1.3’ and others have been associated with the request ‘Request1’.

The request result rank field 840 includes information regarding aranking of a search result(s) associated with a search request. In atleast one embodiment, the request result ID field 835 and the requestresult rank field 840 are linked by for example a pointer. Using theexample illustrated in FIG. 8, the result ‘Result1.1’ has a ranking of‘1’, the result ‘Result1.2’ has a ranking of ‘3’, and the result‘Result1.3’ has a ranking of ‘4’. The content of the request result IDfield 835 and the request result rank field 840 may be used at least inpart to determine an order in which a search result(s) are presented toa user(s) responsive to a search request. Using the example illustratedin FIG. 8, the result ‘Result1.1’ might be preferentially presented to auser(s) responsive to the request ‘Request1’.

The request advertisement ID field 845 includes information of anadvertisement(s) which is associated with a search request. For example,an advertisement may be associated with a search request when a guideselects the advertisement to be presented to a user responsive to thesearch request. For example, if a search request is submitted to thesearch system 130 (FIG. 1), a guide may be selected, and the guide mayprovide an advertisement(s) responsive to the request. Alternately, anadvertisement(s) may be associated with a search request automaticallysuch as when a category(ies) and/or a keyword(s) is associated with thesearch request. Any number of advertisements may be associated with asearch request. Using the example illustrated in FIG. 8, theadvertisements ‘Advert1’; ‘Advert4’, ‘Advert2’, and others have beenassociated with the request ‘Request1’.

The request advertisement rank field 850 includes information regardinga ranking of an advertisement(s) associated with a search request. In atleast one embodiment, the request advertisement ID field 845 and therequest advertisement rank field 850 are linked by for example apointer. Using the example illustrated in FIG. 8, the advertisement‘Advert1’ has a ranking of ‘3’, the advertisement ‘Advert4’ has aranking of ‘1’, and the advertisement ‘Advert2’ has a ranking of ‘5’.The content of the request advertisement ID field 845 and the requestadvertisement rank field 850 may be used at least in part to determinean order in which an advertisement(s) are presented to a user(s)responsive to a search request. Using the example illustrated in FIG. 8,the advertisement ‘Advert4’ might be preferentially presented to auser(s) responsive to the request ‘Request1’.

The request usage information field 855 may include informationregarding a number of times that a search request has been submitted tothe search system 130 (FIG. 1). For example, each time a search requestis submitted to the search system, the usage counter may be updated. Inat least one embodiment, request usage information may be utilized todetermine whether a search result(s) associated with a search requestare to be reviewed. Using the example illustrated in FIG. 8, the request‘Request1’ has ‘use counter=20’, which may indicate that the searchrequest ‘Request1’ has been submitted twenty times.

As illustrated in FIG. 9, an exemplary guide record 900 is provided, ofwhich one or more may be associated with or resident in the searchdatabase 120 (FIG. 1), and may include a guide identification (ID) field905, a guide category ID field 910, a guide topic ID field 915, a guideprofile ID field 920, a guide result ID field 925, a guide communicationinfo field 930, and a guide request ID field 935.

The guide ID field 905 preferably contains a unique identifier of aguide, which is preferably used consistently. For example, in at leastone embodiment, the guide ID field 905 can include a randomly generatednumerical code, and/or a text string indicating a name associated with aguide. A guide ID serves to distinguish the guide record associated witha guide from a guide record associated with other guides. Other uniqueidentifiers of a guide may be utilized without departing from the spiritand scope of the embodiments. Using the example illustrated in FIG. 9,‘Guide1’ is the guide ID associated with the guide record 900.

The guide category ID field 910 includes information of a category(ies)associated with a guide. The content of the guide category ID field 910may be modified based on actions of a guide. A person may select acategory(ies) and/or keyword(s) which is associated with a guide. Acategory may be associated with a guide based on testing of a guide(s).A category may be associated with a guide based on an affiliate groupassociated with a guide. For example, if a guide has chosen to beaffiliated with affiliate groups associated with a type of music, acategory associated with the type of music might be associated with theguide. A category(ies) associated with a guide may be used to selectitems which are to be presented to a guide. A guide(s) may be selectedto vote regarding an item(s) based on the association of a guide with acategory(ies). Using the example illustrated in FIG. 9, the categories‘Category1’ and ‘Category3’ are associated with the guide ‘Guide1’.

The guide topic ID field 915 includes information of a topic(s)associated with a guide. The content of the guide topic ID field 915 maybe modified based on actions of a guide. A person may select a topicwhich is associated with a guide. For example, a guide may select atopic(s) to be associated with the guide during a registration process.A topic may be associated with a guide based on testing of a guide(s).For example, a guide may be required to demonstrate knowledge of acategory and/or topic(s) in order to be associated with the topic(s).The content of the guide topic ID field 915 may be compared to thecontent of a search request in order to determine a ranking of aguide(s) for responding to a search request. Using the exampleillustrated in FIG. 9, the topics ‘reggie miller’, ‘basketball’ and‘fishing’ are associated with the guide ‘Guide1’. This may indicate that‘Guide1’ has indicated an interest in those topics.

The guide profile ID field 920 includes information of a profile(s)associated with a guide. The content of the guide profile ID field 920may be modified based on actions of a guide. A person may select aprofile which is associated with a guide. For example, a guide mayselect a profile(s) to be associated with the guide during aregistration process. A profile may be associated with a guide based ontesting of a guide(s). For example, a guide may be required todemonstrate knowledge relevant to a profile(s) in order to be associatedwith the profile(s), or a guide may take a test which is used togenerate a profile, or a guide may provide information such asdemographic, geographic, personality or other information which may beindicated in a profile(s) associated with the guide. Informationindicated in the content of the guide profile ID field 920 may becompared to information indicated in the content of a profile(s)associated with search request in order to determine a ranking of aguide(s) for responding to a search request. Likewise, a profile(s)associated with a guide(s) may be used to select and/or rank a guide(s)for voting. Using the example illustrated in FIG. 9, the profiles‘Demoprofileg1’, ‘Geoprofileg1’ and ‘Persprofileg1’ are associated withthe guide ‘Guide1’. This may indicate that ‘Guide1’ has indicated and/orgenerated the profiles ‘Demoprofileg1’ which may be a demographicprofile, ‘Geoprofileg1’ which may be a geographic profile and‘Persprofileg1’ which might indicate personality information regardingthe guide ‘Guide1’.

The guide result ID field 925 includes information of a result(s)associated with a guide. The content of the guide result ID field 925may be modified based on actions of a guide. If a guide produces asearch result(s) responsive to a search request(s), an identifier of thesearch result(s) may be included in the guide result ID field 925. Arating(s) and/or ranking(s) associated with a search result(s)associated with a guide may affect compensation for a guide. Likewise ausage indicator associated with a search result(s) provided by a guidemay affect a rating(s) or ranking(s) associated with a guide. Using theexample illustrated in FIG. 9, the results ‘Result1.1’, ‘Result3.1’ and‘Result3.2’ are associated with the guide ‘Guide1’. This may indicatethat ‘Guide1’ has provided the results ‘Result1.1’, ‘Result3.1’, and‘Result3.2’ responsive to a search request(s).

The guide communication info field 930 includes information of adevice(s) and/or service(s) associated with a guide. The content of theguide communication info field 930 may be modified based on actions of aguide. If a guide establishes communications with the search systemusing a device(s) and/or service(s) information regarding the device(s)and/or service(s) may be included in the guide communication info field930. Any type of communication service(s) and/or system(s) may beindicated in the guide communication info field 930. For example, ausername and/or password associated with a guide may be indicated in theguide communication info field 930. Communication services such asInstant Messaging (IM), e-mail, SMS, MMS, EMS, telephone, wireless orwired communication, etc. may be indicated in the guide communicationinfo field 930. A telephone number, an email address, an IM provider andlogin ID, a keyword(s) associated with a service, etc. may be indicatedin the guide communication info field 930, Using the example illustratedin FIG. 9, the login ‘guide1’, the email ‘guide1@chacha.com’, the IMcredential ‘guide1@AIM’ and the phone number ‘317.224.2242’ areassociated with the guide ‘Guide1’. This may indicate that ‘Guide1’ maybe contacted using the login ID ‘guide1’, via email at‘guide1@chacha.com’, via IM as ‘guide1@AIM’ and via voice, text, orother service(s) associated with the phone number ‘317.224.2242’.

The guide request ID field 935 includes information of a request(s)associated with a guide. The content of the guide request ID field 935may be modified based on actions of a guide. If a guide produces asearch result(s) responsive to a search request(s), an identifier of thesearch request(s) may be included in the guide request ID field 935. Arating(s) and/or ranking(s) associated with a search request(s)associated with a guide may affect compensation for a guide. Likewise ausage indicator associated with a search request(s) and a guide mayaffect a rating(s) or ranking(s) associated with the guide. A requestmay be associated with a guide based on a vote(s) cast by a guideregarding items associated with the search request. Using the exampleillustrated in FIG. 9, the requests ‘Request1’, ‘Request3’ and othersare associated with the guide ‘Guide1’. This may indicate that ‘Guide1’has responded to, been selected to respond to, and/or has votedregarding one or more item(s) associated with the requests ‘Request1’and ‘Request3’.

As illustrated in FIG. 10, an exemplary user record 1000 is provided, ofwhich one or more may be associated with or resident in the searchdatabase 120 (FIG. 1), and may include a user identification (ID) field1005, a user profile ID field 1010, a user request ID field 1015, a userresult ID field 1020, a user advertisement ID field 1025, and a usercommunication info field 1030.

The user ID field 1005 preferably contains a unique identifier of auser, which is preferably used consistently. For example, in at leastone embodiment, the user ID field 1005 can include a randomly generatednumerical code, and/or a text string indicating a name associated with auser. A user ID serves to distinguish a user record associated with auser from a user record associated with other users. Other uniqueidentifiers of a user may be utilized without departing from the spiritand scope of the embodiments. In at least one embodiment, a user ID mayinclude a phone number associated with a user. Using the exampleillustrated in FIG. 10, ‘User1’ is the user ID associated with the userrecord 1000.

The user profile ID field 1010 includes information of a profile(s)associated with a user. The content of the user profile ID field 1010may be modified based on actions of a user. A person may select aprofile which is associated with a user. For example, a user may selecta profile(s) to be associated with the user during a registrationprocess. A profile may be associated with a user based on testing of auser(s). For example, a user may be required to demonstrate knowledgerelevant to a profile(s) in order to be associated with the profile(s),or a user may take a test which is used to generate a profile, or a usermay provide information such as demographic, geographic, personality orother information which may be indicated in a profile(s) associated withthe user. A profile(s) associated with a user(s) may be used to selectand/or rank a user(s) for voting. Using the example illustrated in FIG.10, the profiles ‘DemoprofileU1’, ‘GeoprofileU1’ and ‘PersprofileU1’ areassociated with the user ‘User1’. This may indicate that ‘User1’ hasindicated and/or generated the profiles ‘DemoprofileU1’ which may be ademographic profile, ‘GeoprofileU1’ which may be a geographic profileand ‘PersprofileU1’ which might indicate personality informationregarding the user ‘User1’.

The user request ID field 1015 includes information of a request(s)associated with a user. The content of the user request ID field 1015may be modified based on actions of a user. If a user submits a searchrequest to the search system 130 (FIG. 1) an identifier of the searchrequest(s) may be included in the user request ID field 1015. Using theexample illustrated in FIG. 10, the requests ‘Request1’, ‘Request2’ areassociated with the user ‘User1’. This may indicate that ‘User1’ hassubmitted the requests ‘Request1’ and ‘Request2’ to the search system130 (FIG. 1).

The user result ID field 1020 includes information of a result(s)associated with a user. The content of the user result ID field 1020 maybe modified based on actions of a user. If a user receives a searchresult(s) responsive to a search request(s), an identifier of the searchresult(s) may be included in the user result ID field 1020. A usageindicator associated with a search result(s) provided to a user(s) mayaffect a rating(s) or ranking(s) associated with a guide. Using theexample illustrated in FIG. 10, the results ‘Result1.1’, ‘Result1.2’ and‘Result2.1’ are associated with the user ‘User1’. This may indicate that‘User1’ has been presented with the results ‘Result1.1’, ‘Result1.2’,and ‘Result2.1’ responsive to a search request(s).

The user advertisement ID field 1025 includes information of anadvertisement(s) associated with a user. The content of the useradvertisement ID field 1025 may be modified based on actions of a user.If a user receives an advertisement(s) responsive to a searchrequest(s), an identifier of the advertisement(s) may be included in theuser advertisement ID field 1025. A usage indicator associated with anadvertisement(s) provided to a user(s) may affect a rating(s) orranking(s) associated with a guide, compensation for the search system,and/or rating(s) and/or ranking(s) of an advertisement(s). Using theexample illustrated in FIG. 10, the advertisements ‘Advert1’, ‘Advert2’are associated with the user ‘User1’. This may indicate that ‘User1’ hasbeen presented with the advertisements ‘Advert1’, ‘Advert2’.

The user communication info field 1030 includes information of adevice(s) and/or service(s) associated with a user. The content of theuser communication info field 1030 may be modified based on actions of auser. If a user establishes communications with the search system usinga device(s) and/or service(s), information regarding the device(s)and/or service(s) may be included in the user communication info field1030. Any type of communication service(s) and/or system(s) may beindicated in the user communication info field 1030. For example, ausername and/or password associated with a user may be indicated in theuser communication info field 1030. Communication services such asInstant Messaging (IM), e-mail, SMS, MMS, EMS, telephone, wireless orwired communication, etc. may be indicated in the user communicationinfo field 1030. A telephone number, an email address, an IM providerand login ID, a keyword(s) associated with a service, etc. may beindicated in the user communication info field 1030, Using the exampleillustrated in FIG. 10, the login ‘user1’, the email ‘user1@chacha.com’,the twitter account ‘twitter:user1’ and the phone number ‘317.924.2242’are associated with the user ‘User1’. This may indicate that ‘User1’ maybe contacted using the login ID ‘user1’, via email at‘user1@chacha.com’, via Twitter as ‘user1’ and via voice, text, or otherservice(s) associated with the phone number ‘317.924.2242’.

As illustrated in FIG. 11, an exemplary profile record 1100 is provided,of which one or more may be associated with or resident in the searchdatabase 120 (FIG. 1), and may include a profile identification (ID)field 1105, a profile geographic info field 1110, a profile demographicinfo field 1115, a profile personality info field 1120, a profile guideID 1125 and a profile guide rating 1130.

The profile ID field 1105 preferably contains a unique identifier of aprofile, which is preferably used consistently. For example, in at leastone embodiment, the profile ID field 1105 can include a randomlygenerated numerical code, and/or a text string indicating a nameassociated with a profile. A profile ID serves to distinguish a profilerecord associated with a profile from a profile record associated withother profiles. Other unique identifiers of a profile may be utilizedwithout departing from the spirit and scope of the embodiments. In atleast one embodiment, a profile ID may include a description associatedwith a profile. Using the example illustrated in FIG. 11, ‘Profile1’ isthe profile ID associated with the profile record 1100.

The profile geographic info field 1110 may include geographicinformation associated with a profile. Any type of geographicinformation may be indicated in the profile geographic info field 1110.For example, GPS coordinates, a street address, a neighborhood name, apostal code, a city name, a state name, etc. could be indicated in theprofile geographic info field 1110. Using the example illustrated inFIG. 11, the name ‘Indiana’ is associated with the profile ‘Profile1’.This may indicate that guides and/or users with home addresses in thestate of ‘Indiana’ will match the profile ‘Profile1’.

The profile demographic info field 1115 may include demographicinformation associated with a profile. Any type of demographicinformation may be indicated in the profile demographic info field 1115.For example, date of birth, sex, race, political affiliation, income,number of children, marital status, membership in organizations, etc.could be indicated in the profile demographic info field 1115. Using theexample illustrated in FIG. 11, the profile ‘under30’AND‘over20’ isassociated with the profile ‘Profile1’. This may indicate that guidesand/or users with ages greater than 20 years and less than 30 years willmatch the profile ‘Profile1’.

The profile personality info field 1120 may include personalityinformation associated with a profile. Any type of personalityinformation may be indicated in the profile personality info field 1120.For example, information of personality traits such as aggressiveness,curiosity, friendliness, intelligence, honesty, etc. could be indicatedin the profile personality info field 1120. Using the exampleillustrated in FIG. 11, the profile ‘Foodie’ is associated with theprofile ‘Profile1’. This may indicate that guides and/or users with aninterest in food and restaurants will match the profile ‘Profile1’.

The profile guide ID field 1125 includes information of a guide(s) whoare associated with a profile. For example, if a guide elects to receivesearch requests associated with the profile ‘Profile1’, an identifier ofthe guide may be indicated in the profile guide ID field 1125.Information indicated in the profile guide ID field may be used toobtain information associated with a guide using a record(s) such as theguide record 900 (FIG. 9). Using the example illustrated in FIG. 11 theguides ‘Guide1’, ‘Guide3’, ‘Guide2’, ‘Guide4’ and ‘Guide5’ have beenassociated with the profile ‘Profile1’.

The profile guide rating field 1130 includes information regarding arating of a guide(s) associated with a profile. In at least oneembodiment, the profile guide ID field 1125 and the profile guide ratingfield 1130 are linked by for example a pointer. Using the exampleillustrated in FIG. 11, the guide ‘Guide1’ has a rating of ‘2’, theguide ‘Guide3’ has a rating of ‘2’, the guide ‘Guide2’ has a rating of‘1’, the guide ‘Guide4’ has a rating of ‘3’ and the guide ‘Guide5’ has arating of ‘3’. The content of the profile guide ID field 1125 and theprofile guide rating field 1130 may be used at least in part todetermine an order in which a guide(s) are notified responsive to asearch request, a voting activity, or other types of tasks and/oractivities associated with a profile(s). Using the example illustratedin FIG. 11, the guide ‘Guide2’ might be preferentially notified of asearch request associated with the profile ‘Profile1’.

As illustrated in FIG. 12, an exemplary category record 1200 isprovided, of which one or more may be associated with or resident in thesearch database 120 (FIG. 1), and may include a category identification(ID) field 1205, a category description field 1210, a category guide IDfield 1215, a category guide rating field 1220, a category resource IDfield 1225, a category resource rating field 1230, a categoryadvertisement ID field 1235, and a category advertisement rating field1240.

A category record may be created by various processes, including througha guided search session, an interactive training session, anon-interactive training session, a selection(s) made by a guide(s), andmay be imported from an external resource accessible to the searchsystem 130 (FIG. 1), submission of a search request to a resource(s)such as a search engine(s), etc.

The category ID field 1205 preferably contains a unique identifier of acategory, which is preferably used consistently. For example, in atleast one embodiment, the category ID field 1205 can include a randomlygenerated numerical code, and/or a text string indicating the content ofa query. A category ID serves to distinguish the category recordassociated with a category from a category record associated with othercategories. Other unique identifiers of a category may be utilizedwithout departing from the spirit and scope of the embodiments. Asillustrated in FIG. 12, ‘Category1’ is the category ID associated withthe category record 1200 (FIG. 12).

The category description field 1210 includes information of adescription associated with a category. For example, text associatedwith a category may be indicated in the category description field 1210.Such information may be used to assist a guide in associating a searchrequest with a category(ies). In at least one embodiment, a descriptionof a category and other item(s) associated with a category may bepresented to a guide(s) in order to allow a guide(s) to indicate apreferred item(s). For example, an advertisement(s), a searchresource(s), etc. may be presented to a guide(s) associated with acategory(ies) in order that a guide(s) may vote regarding theadvertisement(s), the search resource(s), etc. As illustrated in FIG.12, ‘Entertainment>Dining’ is the category description associated withthe category ‘Category1’.

The category guide ID field 1215 includes information of a guide(s) whoare associated with a category. For example, if a guide elects toreceive search requests associated with the category ‘Category1’, anidentifier of the guide may be indicated in the category guide ID field1215. Information indicated in the category guide ID field may be usedto obtain information associated with a guide using a record(s) such asthe guide record 900 (FIG. 9). Using the example illustrated in FIG. 12the guides ‘Guide1’, ‘Guide2’, and ‘GuideN’ have been associated withthe category ‘Category1’.

The category guide rating field 1220 includes information regarding arating of a guide(s) associated with a category. In at least oneembodiment, the category guide ID field 1215 and the category guiderating field 1220 are linked by for example a pointer. Using the exampleillustrated in FIG. 12, the guide ‘Guide1’ has a rating of ‘10’, theguide ‘Guide2’ has a rating of ‘6’, and the guide ‘GuideN’ has a ratingof ‘2’. The content of the category guide ID field 1215 and the categoryguide rating field 1220 may be used at least in part to determine anorder in which a guide(s) are notified responsive to a search request, avoting activity, or other types of tasks and/or activities associatedwith a category(ies). Using the example illustrated in FIG. 12, theguide ‘Guide1’ might be preferentially notified of a search requestassociated with the category ‘Category1’.

The category resource ID field 1225 includes information of aresource(s) associated with a category. For example, if a guide utilizesa resource to respond to a search request(s) associated with thecategory ‘Category1’, an identifier of the resource may be indicated inthe category resource ID field 1225. Information indicated in thecategory resource ID field may be used to obtain information associatedwith a resource using a record(s) such as the resource record 1400illustrated in FIG. 14. Using the example illustrated in FIG. 12 theresources ‘Resource1’, ‘Resource2’, and ‘ResourceN’ have been associatedwith the category ‘Category1’.

The category resource rating field 1230 includes information regarding arating of a resource(s) associated with a category. In at least oneembodiment, the category resource ID field 1225 and the categoryresource rating field 1230 are linked by for example a pointer. Usingthe example illustrated in FIG. 12, the resource ‘Resource1’ has arating of ‘8’, the resource ‘Resource2’ has a rating of ‘8.5’, and theresource ‘ResourceN’ has a rating of ‘3’. The content of the categoryresource ID field 1225 and the category resource rating field 1230 maybe used at least in part to determine an order in which a searchresource(s) are presented and/or utilized responsive to a search requestassociated with a category(ies). Using the example illustrated in FIG.12, the resource ‘Resource2’ might be preferentially presented to aguide(s) associated with the category ‘Category1’.

The category advertisement ID field 1235 includes information of anadvertisement(s) associated with a category. For example, if a guideselects an advertisement to be sent to a user responsive to a searchrequest(s) associated with the category ‘Category1’, an identifier ofthe advertisement may be indicated in the category advertisement IDfield 1235. Information indicated in the category advertisement ID fieldmay be used to obtain information associated with an advertisement usinga record(s) such as the advertisement record 1500 illustrated in FIG.15. Using the example illustrated in FIG. 12 the advertisements‘Advert1’, ‘Advert2’, and ‘AdvertN’ have been associated with thecategory ‘Category1’.

The category advertisement rating field 1240 includes informationregarding a rating of an advertisement(s) associated with a category. Inat least one embodiment, the category advertisement ID field 1235 andthe category advertisement rating field 1240 are linked by for example apointer. Using the example illustrated in FIG. 12, the advertisement‘Advert1’ has a rating of ‘1’, the advertisement ‘Advert2’ has a ratingof ‘3’, and the advertisement ‘AdvertN’ has a rating of ‘9’. The contentof the category advertisement ID field 1235 and the categoryadvertisement rating field 1240 may be used at least in part todetermine an order in which an advertisement(s) are presented to aguide(s) and/or a user(s) responsive to a search request associated witha category(ies). Using the example illustrated in FIG. 12, theadvertisement ‘AdvertN’ might be preferentially presented to a guide(s)and/or a user(s) responsive to a search request associated with thecategory ‘Category1’.

Rating information of a guide(s), an advertisement(s), a searchresource(s), and other information associated with a category may bebased at least in part on votes of guides associated with the category,but may be based on any information indicated in the database 120 (FIG.1).

As illustrated in FIG. 13, an exemplary result record 1300 is provided,of which one or more may be associated with or resident in the searchdatabase 120 (FIG. 1), and may include a result identification (ID)field 1305, a result guide ID field 1310, a result content info field1315, a result resource ID field 1320 and a result usage informationfield 1325.

The result ID field 1305 preferably contains a unique identifier of aresult, which is preferably used consistently. For example, in at leastone embodiment, the result ID field 1305 can include a randomlygenerated numerical code, and/or a text string indicating a nameassociated with a result. A result ID serves to distinguish the resultrecord associated with a result from a result record associated withother results. Other unique identifiers of a result may be utilizedwithout departing from the spirit and scope of the embodiments. In atleast one embodiment, a result ID may include a description associatedwith a result. Using the example illustrated in FIG. 13, ‘Result1.1’ isthe result ID associated with the result record 1300.

The result guide ID field 1310 may include information of a guide(s)associated with a result. For example, a guide who obtained a searchresult may be indicated in the result guide ID field 1310. In at leastone embodiment a guide ID associated with a guide(s) who have voted on aresult may be indicated in the result guide ID field 1310. Informationcontained in the result guide ID field 1310 may be used to obtaininformation regarding a guide associated with a search result based onrecords such as the guide record 900 (FIG. 9). Using the exampleillustrated in FIG. 13, the guide ID ‘Guide1’ is associated with theresult ‘Result1.1’ which may indicate that the guide ‘Guide1’ hasprovided the search result ‘Result1.1’.

The result content info field 1315 may include information of content ofa search result. For example, a text snippet associated with a searchresult may be indicated in the result content info field 1315. A URL orother information associated with a search result may be indicated inthe result content info field 1315. Information contained in the resultcontent info field 1315 may be used to provide a search result to auser(s) and/or a guide(s). Using the example illustrated in FIG. 13, thetext snippet ‘That sensations is the best That restaurant in Indy’ isassociated with the result ‘Result1.1’. This may indicate that the textsnippet may be provided to a user when the result ‘Result1.1’ isselected to be presented to a user.

The result resource ID field 1320 may include information of a searchresource associated with a search result. For example, a URL associatedwith a web page from which a text snippet was obtained may be indicatedin the result resource ID field 1320. A URL or other information such asa search query and/or a macro instruction(s) associated with a searchresource may be indicated in the result resource ID field 1320.Information contained in the result resource ID field 1320 may be usedto provide access to a search resource(s) for a guide(s) and/or thesearch system 130 (FIG. 1). Using the example illustrated in FIG. 13,‘Resource2’ is the resource ID associated with the result ‘Result1.1’.This may indicate that the resource ‘Resource2’ was used to obtain thesearch result ‘Result1.1’.

The result usage information field 1325 may include informationregarding the use of a search result. For example, a number of timesthat a search result has been presented to a user(s) may be indicated inthe result usage information field 1325. Using the example illustratedin FIG. 13, the indicator ‘use counter=120’ is associated with theresult ‘Result1.1’. This may indicate that the result ‘Result1.1’ hasbeen presented to a user(s) one hundred twenty times. Information in theresult usage information field 1325 may be utilized to determine whethera search result is to be reviewed.

As illustrated in FIG. 14, an exemplary resource record 1400 isprovided, of which one or more may be associated with or resident in thesearch database 120 (FIG. 1), and may include a resource identification(ID) field 1405, a resource guide ID field 1410, a resource descriptionfield 1415, a resource access info field 1420 and a resource usageinformation field 1425.

The resource ID field 1405 preferably contains a unique identifier of aresource, which is preferably used consistently. For example, in atleast one embodiment, the resource ID field 1405 can include a randomlygenerated numerical code, and/or a text string indicating a nameassociated with a resource. A resource ID serves to distinguish theresource record associated with a resource from a resource recordassociated with other resources. Other unique identifiers of a resourcemay be utilized without departing from the spirit and scope of theembodiments. In at least one embodiment, a resource ID may include adescription associated with a resource. Using the example illustrated inFIG. 14, ‘Resource2’ is the resource ID associated with the resourcerecord 1400.

The resource guide ID field 1410 may include information of a guide(s)associated with a resource. For example, a guide(s) who utilized asearch resource may be indicated in the resource guide ID field 1410. Inat least one embodiment a guide ID associated with a guide(s) who havevoted on a resource may be indicated in the resource guide ID field1410. Information contained in the resource guide ID field 1410 may beused to obtain information regarding a guide associated with a searchresource based on records such as the guide record 900 (FIG. 9). Usingthe example illustrated in FIG. 14, the guide ID ‘Guide1’ is associatedwith the resource ‘Resource2’ which may indicate that the guide ‘Guide1’has administrative access to the search resource ‘Resource2’.

The resource description field 1415 may include information of adescription of a search resource. For example, a text snippet associatedwith a search resource may be indicated in the resource descriptionfield 1415. A URL or other information associated with a search resourcemay be indicated in the resource description field 1415. Informationcontained in the resource description field 1415 may be used to providea description of search resource to a user(s) and/or a guide(s). Usingthe example illustrated in FIG. 14, the text snippet ‘Dining in Indiana’is associated with the resource ‘Resource2’.

The resource access info field 1420 may include access informationassociated with a search resource. For example, a URL associated with aweb page may be indicated in the resource access info field 1420. Apassword, a login ID, a template for submitting a search query, an APIfor submitting and receiving information or any other information whichmay be utilized to access a search resource may be indicated in theresource access info field 1420. Information contained in the resourceaccess info field 1420 may be used to provide access to a searchresource(s) for a guide(s) and/or the search system 130 (FIG. 1). Usingthe example illustrated in FIG. 14, the URL ‘www.diningindy.com’ is theaccess information associated with the resource ‘Resource2’. This mayfor example indicate that a search result(s) may be obtained using theresources found at ‘www.diningindy.com’.

The resource usage information field 1425 may include informationregarding the use of a search resource. For example, a number of timesthat a search resource has been utilized by a guide(s) may be indicatedin the resource usage information field 1425. Using the exampleillustrated in FIG. 14, the indicator ‘use counter=180’ is associatedwith the resource ‘Resource2’. This may indicate that the resource‘Resource2’ has been utilized by a guide(s) one hundred eighty times.Information in the resource usage information field 1425 may be utilizedto determine whether a search resource is to be reviewed.

As illustrated in FIG. 15, an exemplary advertisement record 1500 isprovided, of which one or more may be associated with or resident in thesearch database 120 (FIG. 1), and may include an advertisementidentification (ID) field 1505, an advertisement description field 1510,an advertisement access info field 1515 and an advertisement usageinformation field 1520.

The advertisement ID field 1505 preferably contains a unique identifierof an advertisement, which is preferably used consistently. For example,in at least one embodiment, the advertisement ID field 1505 can includea randomly generated numerical code, and/or a text string indicating aname associated with an advertisement. An advertisement ID serves todistinguish the advertisement record associated with an advertisementfrom an advertisement record associated with other advertisements. Otherunique identifiers of an advertisement may be utilized without departingfrom the spirit and scope of the embodiments. In at least oneembodiment, an advertisement ID may include a description associatedwith an advertisement. Using the example illustrated in FIG. 15,‘Advert1’ is the advertisement ID associated with the advertisementrecord 1500.

The advertisement description field 1510 may include information of adescription associated with an advertisement. For example, a descriptionof an advertisement may be presented to a guide(s) in order that a guidemay select an advertisement to be associated with a search request.Using the example illustrated in FIG. 15, ‘Gokimchee the asian groceryin fishers’ is the advertisement description associated with theadvertisement ‘Advert1’. Any type of information such as text, audio,video, images, etc. may be indicated in the advertisement descriptionfield 1510.

The advertisement access info field 1515 may include access informationassociated with an advertisement. For example, a URL associated with aweb page may be indicated in the advertisement access info field 1515.Information contained in the advertisement access info field 1515 may beused to provide access to an advertisement(s) for a guide(s) and/or thesearch system 130 (FIG. 1). Using the example illustrated in FIG. 15,the URL ‘https://adserverchacha.com\gokimchee’ is the access informationassociated with the advertisement ‘Advert1’. This may for exampleindicate that the advertisement ‘Advert1’ may be delivered to a userusing content accessed at ‘https://adserverchacha.com\gokimchee’.

The advertisement usage information field 1520 may include informationregarding the use of an advertisement. For example, a number of timesthat an advertisement has been delivered to a user(s) may be indicatedin the advertisement usage information field 1520. Using the exampleillustrated in FIG. 15, the indicator ‘use counter=180’ is associatedwith the advertisement ‘Advert1’. This may indicate that theadvertisement ‘Advert1’ has been delivered to user(s) one hundred eightytimes. Information in the advertisement usage information field 1520 maybe utilized to determine whether an advertisement is to be reviewed, andmay be utilized to determine compensation for a guide(s) and/or theprovider of the search system 130 (FIG. 1).

As illustrated in FIG. 16, a process 1600 for creating a notificationtable is provided. A notification table may be used to determine anorder in which one or more persons and/or entities may be notified of atask.

In operation 1605 a determination is made as to whether a notificationtable preparation task is required. If it is determined in operation1605 that a notification table is not to be prepared, control remains atoperation 1605 and process 1600 continues. If in operation 1605 it isdetermined that a notification table is to be prepared, control ispassed to operation 1610 and process 1600 continues. In at least oneembodiment, the determination that a notification table is to beprepared is based on a task becoming available. For example, if a searchrequest is submitted, or if a review is needed, or if any other type oftask requiring work to be performed is identified, it may be determinedthat a notification table is to be prepared. In at least one embodiment,if all resources associated with a notification table have beennotified, it may be determined that a notification table is to beprepared.

In operation 1610, a reference item(s) associated with task is obtained.For example, information of a search request may be obtained, which mayinclude a query, image(s), audio, media, keyword(s), category(ies),tag(s), profile(s), etc. Likewise, information of a category, anadvertisement, or any item(s) which may be associated with informationwhich may be associated with rating information may be obtained. Controlis passed to operation 1615 and process 1600 continues.

In operation 1615, rating and/or ranking information associated with areference item(s) is obtained. For example, rating information of aperson, resource, or entity associated with a reference item, which mayinclude a query, image(s), audio, media, keyword(s), category(ies),tag(s), profile(s), etc. may be obtained. A rating(s) may be based onany type of information. For example, a guide rating associated with acategory(ies), a keyword(s), a profile(s), etc. may be based on userratings, peer ratings, ratings of results produced by the guide,activity(ies) of a guide, training of a guide, etc. Any number of itemsassociated with rating information may be used to identify ratinginformation associated with a guide. While a guide has been used for thepurpose of simplicity, users, resource(s) and/or other items may beassociated a reference item which may be associated with ratinginformation. Control is passed to operation 1620 and process 1600continues.

In operation 1620, a ranking of a guide(s) and/or other items identifiedin operation 1615 is performed. For example, all guides who are eligibleand/or available for an activity may be ranked based on the rating dataidentified in operation 1615. Likewise, users who might be availableand/or eligible to perform an activity might be ranked, or resourceswhich might respond to a request might be ranked, or resources whichmight provide information of any sort might be ranked. Any type ofitem(s) indicated in the database 120 (FIG. 1) might be ranked based onrating information. Control is passed to operation 1625 and process 1600continues.

In operation 1625, the system database 120 is updated. In at least oneembodiment, a notification table is made available for use by the system130 (FIG. 1). Information associated with a notification table may berecorded, such as a task ID associated with a notification table, etc.Information associated with items indicated in a notification table maybe updated to indicate that an item has been added to a notificationtable, or that an item was not included in a notification table, or aranking associated with an item in a notification table, etc. Control ispassed to operation 1605 and process 1600 continues.

A database relationship for creating a ranked list of guides to beassociated with a reference item and a review items is illustrated inFIG. 16A. A reference item may be an item such as a search request, acategory, or any combination of item(s) which may be presented to aguide as a reference for an action. For example, a search request may bepresented to a guide in order that a guide may provide a searchresult(s) responsive to a search request, or a search request may bepresented to a guide in order that a guide may vote regarding searchresult(s) associated with the search request, or an indicator of acategory may be presented to a guide in order that a guide may voteregarding a search resource(s), a topic, or a related categoryassociated with the category. A review item may be an item such as asearch result, a search resource, an advertisement, or any other itemwhich may be presented for rating or voting. An index item may be anyitem such as a category, a profile, a location, a keyword, etc. which isused to select and/or rank guides based on a rating or ranking of guidesassociated with the guides and the index item. An index item may beassociated with a reference item.

A notification table such as the notification table 1650 illustrated inFIG. 16A may be created of which one or more may be associated with orresident in the search database 120 (FIG. 1), and may include anotification table ID field 1652, a notification table guide ID field1655, a notification table guide ranking field 1660, a notificationindicator field 1665 and an availability indicator field 1670. A tasknotification table may be associated with a reference item such as asearch request.

The notification table ID field 1652 preferably contains a uniqueidentifier of a notification table, which is preferably usedconsistently. For example, in at least one embodiment, the notificationtable ID field 1652 can include a randomly generated numerical code,and/or a text string indicating a name associated with a task. Anotification table ID serves to distinguish the notification tablerecord associated with a notification table from a notification tablerecord associated with other notification tables. Other uniqueidentifiers of a notification table may be utilized without departingfrom the spirit and scope of the embodiments. In at least oneembodiment, a notification table ID may include a description associatedwith a task. Using the example illustrated in FIG. 16A,‘Review1Request4’ is the notification table ID associated with thenotification table 1650.

The notification table guide ID field 1655 includes a list of guideidentifiers which may be used to contact a guide(s), the notificationtable guide ranking field 1660 indicates a ranking of the guideassociated with the guide ID indicated in the notification table guideID field 1655. Using the example in FIG. 16, ‘Guide2’ is ranked ‘1’,‘Guide1’ is ranked ‘2’, ‘Guide3’ is ranked ‘3’, ‘Guide5’ is ranked ‘4’,and ‘Guide4’ is ranked ‘5’. The notification indicator field 1665indicates whether a guide indicated in the notification table guide IDfield 1655 has been notified of a task. Using the example in FIG. 16A,none of the guides has been notified of the task ‘Review1Request4’ asindicated by the ‘No’ in the notification indicator field 1665. Theavailability indicator field 1670 indicates the availability of a guide.The availability status of a guide may change from time to time asguides become available from a task, as guides login to the system 100(FIG. 1), or as guides are assigned tasks. Using the example in FIG.16A, the guides ‘Guide1’, ‘Guide3’ and ‘Guide4’ are available asindicated by the ‘Yes’ in the availability indicator field 1670.

In order to determine the ranking of guides as indicated in thenotification table guide ranking field 1660 of the notification table1650, the rating(s) and/or ranking(s) of a guide(s) associated with oneor more index items may be utilized. Using the example illustrated inFIG. 16A, a search ‘Request4’ associated with the request record 800 aas indicated in the request ID field 805 a, is associated with thereview items ‘Result4.1’, ‘Result4.2’, and ‘Result4.3’ as indicated inthe request result ID field 835 a. The request ‘Request 4’ is associatedwith the categories ‘Category1’ and ‘Category2’ as indicated in therequest category ID field 825 a, and the profile ‘Profile1’ as indicatedin the request profile ID field 830 a. A ranking of guides is performedbased on information contained in the category records 1200 a and 1200b, and the profile record 1100 a. For example, a geographic profilewhich includes a location may be associated with a search request, and aguide within a first distance of the location may be given a rating ‘1’,guides within a second distance of the location may receive a rating of‘2’, and guide at any greater distance may receive a rating of ‘3’. Aguide(s) may have a rating(s) associated with a category. For example, aguide rating may be based on a rating(s) of search results produced by aguide for search requests associated with a category, wherein a higherrating indicates a better success rate. Using the example illustrated inFIG. 16A, the guide ranking is based first on the rating associated withgeographic location indicated in ‘Profile1’ and second on the average ofrating(s) associated with ‘Category1’ and ‘Category2’. Continuing withthe example in FIG. 16A, the guide with the lowest rating with respectto the profile ‘Profile 1’ (i.e. ‘Guide 2’) is ranked first due togeographic proximity even though ‘Guide2’ has a low average ratingassociated with ‘Category1’ and ‘Category2’. The guides ‘Guide1’ and‘Guide3’ are ranked ‘2’ and ‘3’, respectively, as they have the samerating with respect to ‘Profile1’ and ‘Guide1’ has a higher averagerating associated with ‘Category1’ and ‘Category2’, and ‘Guide5’ and‘Guide4’ are ranked ‘4’ and ‘5’, respectively, as they are rated ‘3’with respect to ‘Profile1’ and ‘Guide5’ has a higher average ratingassociated with ‘Category1’ and ‘Category2’.

While a few index items are illustrated in FIG. 16A, no limitation isimplied thereby. Any number of index items may be utilized and anynumber of guides may be ranked using the method 1600 (FIG. 16). Anycriteria may be applied to determine a ranking and/or rating associatedwith an index item such as a category, a location, a keyword, a profile,a tag, a topic, etc. While a group of guides who may vote regarding asearch result(s) associated with a search request is used for thepurposes of illustration, no limitation is implied thereby. For example,a group of users might be selected to review a search result(s)associated with a search request, or a group of guides might be notifiedof a search request which is pending a response. A notification tablemay be used to notify any person(s) or entity(ies) of a pending task(s).

As illustrated in FIG. 17, a process 1700 for notification is provided.

In operation 1705, a determination is made as to whether review of anitem is needed. If it is determined in operation 1705 that review of anitem is not needed, control remains at operation 1705 and process 1700continues. If in operation 1705 it is determined that review of an itemis needed, control is passed to operation 1710 and process 1700continues. In at least one embodiment, a value of a usage countassociated with an item is utilized to determine whether review of anitem is needed. In at least one embodiment, a number of review itemsassociated with a reference item is utilized to determine whether reviewof an item is needed.

In operation 1710, review items associated with a reference item areobtained. For example if a review item such as a search result or asearch resource has been determined to need review, a reference itemsuch as a search query or a category may be presented to a searcher inorder that an additional review item may be associated with a referenceitem. Likewise if a number of search results have been associated with asearch request, or a number of topics have been associated with acategory, a list of the reference items may be compiled for review.Control is passed to operation 1715 and process 1700 continues.

In operation 1715, a list of reviewers associated with a reference itemis created. Voters associated with a reference item such as a searchrequest may be identified. For example, voters associated with topic(s)and/or keyword(s) of a search request may be added to a reviewer list,or voters associated with a category(ies) and/or a profile(s) may beadded to a reviewer list. Any information regarding a reviewer or votermay be used to select members of a list of reviewers for a referenceitem. For example, if a list of reviewers has been previously compiledbased on one or more criteria, a list of reviewers may be compiled basedon different criteria. Control is passed to operation 1720 and process1700 continues.

In operation 1720, a ranking of reviewers is performed. For example, anotification table such as the notification table 1650 (FIG. 16A) may beconstructed. Reviewers may be ranked based on various criteria. Forexample, reviewers may be ranked based on a rating(s) and/or ranking(s)associated with a reference item, as illustrated in FIG. 16A. In atleast one embodiment, reviewers may be ranked based at least in part ona vote weight associated with a reviewer. Control is passed to operation1725 and process 1700 continues.

In operation 1725, a reviewer(s) are notified in an order determined bythe reviewer's rank. The highest ranking reviewer who is currentlyavailable and has not been notified of a review task is notified.Control is passed to operation 1730 and process 1700 continues.

In operation 1730 review information is obtained from a reviewer(s). Forexample one or more voters may register a vote regarding a reviewitem(s), a reviewer may indicate acceptance or rejection of a reviewitem(s), etc, Control is passed to operation 1735 and process 1700continues.

In operation 1735, a determination is made as to whether a review iscomplete. If it is determined in operation 1735 that a review iscomplete, control is passed to operation 1745 and process 1700continues. If in operation 1735 it is determined that a review is notcomplete, control is passed to operation 1740 and process 1700continues. Various criteria may be used to determine whether a review iscomplete. Completion of a process such as the process 300 (FIG. 3) orthe process 600 (FIG. 6) or the processes illustrated in FIG. 7C, 7D, or7E may be used to determine that a review is complete. A review may bedetermined to be complete if a time period between the start of a reviewand the current time has exceeded a value.

In operation 1740, a determination is made as to whether all availablereviewers in the review list created in operation 1715 have beennotified. If it is determined in operation 1740 that all availablereviewers in the review list created in operation 1715 have not beennotified, control is passed to operation 1725 and process 1700continues. If in operation 1740 it is determined that all availablereviewers in the review list created in operation 1715 have beennotified, control is passed to operation 1715 and process 1700continues.

In operation 1745, the search database 120 (FIG. 1) is updated.Information of a rating and/or vote weight associated with a voter(s)and/or a profile(s), a category(ies), keyword(s), etc. is recorded.Information of a rating(s) and/or ranking(s) associated with item(s)such as a search result(s), a search resource(s), an advertisement(s),etc. is recorded and updated. Control is passed to operation 1705 andprocess 1700 continues.

FIG. 17A illustrates a notification process utilizing a notificationtable such as the notification table 1650 a. A notification table ID isassigned to the notification table as indicated in the notificationtable ID field 1652 a of the notification table 1650 a. When thenotification table 1650 a is created, the notification indicator field1665 a, and the availability indicator field 1670 a indicate thenotification status and the availability status respectively of theselected reviewers at an initial time, t1. Likewise, the guide ID andthe guide rank are indicated in the notification table guide ID field1655 a and the notification table guide ranking field 1660 a.

As guides are notified of a task, the notification indicator field 1665a is modified to indicate the notification status of guides at a latertime, t2. As illustrated in FIG. 17A, the notification status of theguides ‘Guide2’ and ‘Guide3’ have been set to ‘Yes’ to indicate that‘Guide2’ and ‘Guide3’ have been notified of a task. As each guide may beallowed a time interval in which to respond to a task, a lower rankingguide such as ‘GuideN’ may not be notified at the time t2 although theguide is available for the task ‘Review1Request2’.

As the availability status of guides may change, the guide ‘Guide1’ maybecome available at a time t3 which is later than t2 as indicated in theavailability indicator field 1670 a. In such a case, the guide ‘Guide1’will be notified before the guide ‘GuideN’ due to the higher ranking of‘Guide1’.

At some later time, t4, all available guides may have been notified, asindicated in the notification indicator field 1665 a of the notificationtable 1650 a. In such a case, a new notification table 1650 b may becreated as indicated by the notification table ID field 1652 b. Thenotification table guide ID field 1655 b may include one or more guideswho have previously been notified, as indicated in the notificationindicator field 1665 b. A guide who has been previously notified may notbe notified again. A higher ranking guide as indicated in thenotification table guide ranking field 1660 b such as the guide ‘Guide0’who has been included in the notification table 1650 b may be notifiedfirst. A lower ranking guide such as the guide ‘Guide4’ who has becomeavailable as indicated in the availability indicator field 1670 b andwas not previously notified as indicated in the notification indicatorfield 1665 b may not be notified until higher ranking guides such as‘Guide0’ and ‘Guide5’ have been notified of a task associated with thenotification tables 1650 a and 1650 b. The persistence of thenotification indicator associated with a task which is associated with anotification table may improve the quality of the guides, who may beguides or other voters who are assigned to a task while avoidingnotifying the same guide multiple times of a task.

As illustrated in FIG. 18A, an exemplary GUI is provided for selectionof result(s) associated with a query, resources associated with a branchof a taxonomy, or any review items(s) associated with a referenceitem(s). The GUI 1800 includes selection filtering controls 1805, aselection display window 1810, a rate/recommend button 1812, selectiontype indicators 1815, a scorecard indicator window 1820, a close button1825, an availability selection control 1830, a logoff control 1835, andactivity selection tabs 1840. The GUI 1800 may be used to selectresult(s) and/or other items to rate and/or edit. Items may be selectedusing filtering criteria. For example, queries may be sorted by mostpopular queries, most recent queries, queries which have a specificnumber of votes, queries associated with a profile, queries associatedwith a category or taxonomy branch, etc. A profile might includeinformation such as a time interval, geographic location, or any otheritem(s) which might be used to filter queries based on informationindicated in the search database 120 (FIG. 1). In at least oneembodiment, a guide is able to indicate filtering criteria via a listthat is provided as a drop-down menu, or by typing one or more words ina text entry box, etc.

The filter controls 1805 may be used to select a query(ies) usinginformation associated with a query, such as a category(ies),keyword(s), tag(s), etc. Using the example illustrated in FIG. 18, thequery type selector 1805 a may be used to select requests or queries bypopularity, time submitted, or other filtering options indicated in adrop-down list and/or typing box. The phrase selection filter control1805 b may be used to select queries which contain one or more wordsand/or phrases, and may be provided as a drop-down list and/or typingbox. The category selection filter control 1805 c may be used to selectone or more categories which may be associated with a search request orsearch query. Various other forms of filtering interface may also beused.

In at least one embodiment, the filter controls 1805 include checkboxesfor selecting a variety of selection options. The filter selectioncontrol 1805 d may be used to select queries which are under review byselecting the check-box indicated. The filter selection control 1805 emay be used to select queries which are eligible for review by selectingthe check-box indicated. The filter selection control 1805 f may be usedto select queries which are associated with one or more results byselecting the check-box indicated. The filter selection control 1805 gmay be used to select queries which are not associated with a result byselecting the check-box indicated. Other viewing options might bepresented, such as displaying only queries with recommendations, querieswith no recommendations, queries with automated results, queries withhuman selected results, etc. Any filtering options may be selected andalternate filtering mechanisms and criteria may be provided in the GUI1800, and thus the disclosure is not limited to only the optionsmentioned.

If a filtering method is selected, the selection display window 1810displays the selected query(ies). In at least one embodiment the GUI1800 includes an indicator(s) to designate whether a result is availablefor rating, for rating and recommending, and/or for recommending newresults, or other activities. Alternate indications and/or multipleindications might be presented without departing from the spirit of theembodiments disclosed herein. In at least one embodiment, the indicatorsare “flags” of different colors to indicate the status of a query and/orsearch results such as the selection type indicators 1815. Using theexample illustrated in FIG. 18A, the selection type indicator 1815 a maybe used to indicate that a selected search query is available for votingand recommending a search result. The selection type indicator 1815 bmay be used to indicate that a search query is available forrecommending a result. Other types of indicators and/or otherdesignations for the results may be provided, and the GUI 1800 is notlimited to the particular embodiment shown. For example, queries withresults from a particular source, or queries with results from multiplesources, etc. might be indicated using an indicator such as theselection type indicators 1815 a and 1815 b. Multiple indicators mightbe associated with a query(ies).

A query and associated results for review may be chosen from theselection window 1810 by “clicking” on or otherwise selecting thedisplayed item. If an item(s) is selected in the selection displaywindow 1810, the rate/recommend button 1812 may be used to initiate arating and/or review session. If an item is selected a GUI such as theGUI 1850 illustrated in FIG. 18B may be displayed which may be used torate, and/or otherwise indicate preferences for an item(s) associatedwith a selected query. In at least one embodiment, a query and/orresult(s), etc. may be selected without guide actions. The search system130 (FIG. 1) may make a selection based on any information indicated inthe search database 120.

The close button 1825 may be used to close a session and return to otheractivities. The log-off control 1835 may be used to log off of thesearch system 130 (FIG. 1).

The scorecard indicator 1820 (FIG. 18A) allows a user of the GUI 1800 toview information related to his or her accumulation of ‘points’ whichmay be related to compensation and/or rating, etc. of a guide orPaidSearcher™. In at least one embodiment, a total available points,points reserved (i.e., in suspense for any reason), points available(which might be put at risk via for example, a competition provided bythe provider of the search system 130 (FIG. 1), the outcome of a ratingsession, etc. or any mechanism), a points/recommendation (number ofpoints put ‘at risk’ via adding a result and/or resource for vetting byother guides) and an indicator of available recommendation ‘tickets’available may be displayed. In this way a guide or user of the GUI 1800is readily apprised of his or her point totals regardless of theactivity selected.

The GUI 1800 (FIG. 18A) may include one or more selection tabs such asselection tabs 1840. The selection tabs 1840 may be used to selectactivities, and/or to review other items. Using the example illustratedin FIG. 18, the ‘Rate and Review Searches’ tab 1840 b may cause the GUI1800 to be displayed, and the ‘Review Qualified Keywords’ tab 1840 a maycause a GUI such as GUI the 1980 illustrated in FIG. 18F to bepresented.

The availability selection control 1830 (FIG. 18A) may be used toindicate a status. In at least one embodiment a guide may indicate hisor her availability to accept queries from users, and/or availabilityfor other types of activity. As shown in FIG. 18A, a drop-down menu maybe provided from which a guide may for example elect to be ‘available’for live queries and/or voice queries, but ‘not available’ for trainingother guides in an interactive session. Alternate options and/orselection techniques might be employed within the spirit of thisdisclosure.

As illustrated in FIG. 18B, an exemplary GUI 1850 is shown for ratingand/or voting on result(s) associated with a query and/or a profile, acategory(ies), etc. and/or recommending new result(s) to be associatedwith a query. For example, a query record(s) 800 (FIG. 8) or categoryrecord(s) 1200 (FIG. 12) might be selected using a GUI such as GUI 1800(FIG. 18A). The GUI 1850 (FIG. 18B) includes a scorecard indicator 1820,a search query box 1855, a recommend button 1860, a done button 1865, aselection tabs 1870, viewing controls 1875, a filter control 1880, asubmit button 1885, an item display window 1890, a browser window 1895,voting controls 1897 and a new category button 1898.

The functionality of the scorecard indicator 1820 was previouslydescribed with respect to FIG. 18A. The search query box 1855 maycontain information of a query which is associated with a result(s) tobe reviewed by a user of the GUI 1850. The search query box 1855 mayserve as an aid as to the objective of the rating session. While the GUI1850 is illustrated using a text query, other media, such as an image, amedia clip, or a combination of media, etc. might be presented withinthe spirit of the disclosure herein.

The recommend button 1860 may be used to elect to associate one or moreresults with a query. In at least one embodiment, activation of therecommend button 1860 may cause a GUI such as the GUI 1900 illustratedin FIG. 18C to be displayed.

The ‘I'm Done’ button 1865 (FIG. 18B) may be used to submit a vote(s)and/or recommendation(s) for use by the search system 130 (FIG. 1). Forexample, a vote(s) may be recorded in the search system database 120,and/or a recommended search result may be associated with a query by forexample being added to the results field of a query record (e.g., therequest record 800 (FIG. 8)).

The selection tabs 1870 may be used to select additional GUI's which maydisplay information regarding a session. In the example illustrated inFIG. 18B, the selection tabs 1870 may display a voting GUI such as theGUI 1850, a list of results recommended in session(s), and a summary ofvoting and recommendation activity performed in a current session.

The viewing controls 1875 may be used to control the informationdisplayed in the item display window 1890. In the example illustrated inFIG. 18B, the ‘expand all’ button 1875 a allows the user to view asummary and a URL indicator for each result. The ‘collapse all’ button1875 b makes each result appear as a label. The ‘expand all details’button 1875 c allows the user to see the URL, a comment added by a guideor author of the result, and other information associated with theresult. The ‘collapse all details’ button 1875 d collapses the detailsassociated with a result and displays only the label. Any individualresult may be independently collapsed or expanded using viewingcontrols, such as the viewing controls 1875 e and 1875 f, in the displaywindow 1890. Activating the viewing control 1875 e may cause the answerdetails to be toggled between being displayed or not. Activating theviewing control 1875 f may toggle between displaying detail associatedwith a result and not.

The filter control 1880 may be used to alter the filtering criteriaapplied to a query. For example, an alternate profile or categorizationof a query might be selected in order to view results associated with aquery which has been categorized according to an alternate intent. Usingthe example illustrated in FIG. 18B, the query ‘home made nail polishremover’ might be categorized under ‘Health>Beauty’, and a result(s)associated with it based on that categorization. However, otherresult(s) might be associated with a query (i.e., ‘home made nail polishremover’) categorized as ‘Science>Chemistry’. In the former instance aresult(s) might relate to common household products which could be usedas a nail polish remover whereas in the latter a result(s) might discussthe chemicals involved and their properties. The filter control 1880 maybe implemented as a drop-down menu, a typing box, etc.

The item display window 1890 may be used to display information relatingto results which have been associated with a query. In particular,results which are associated with the query string in the search querybox 1855 and filtered per the selection made using the filter control1880. The item display window 1890 may further allow a user to registeropinions about a result(s) group displayed in the display window. Thevoting controls 1897 allow a user to select a best result from agroup(s) of results appearing in the item display window 1890. In theexample illustrated in FIG. 18B, the ‘A/B’ buttons allow a user toselect a result labeled ‘A’ as the superior result compared to a resultlabeled ‘B’, or vice versa. Alternately, a user may elect to use acontrol such as the ‘propose new category’ button 1898 to propose analternate categorization of a result group which might make a result(s)more relevant in response to a query indicated in the query box 1855. Anaction such as activating the ‘propose new category’ button 1898 maycause a GUI such as the GUI 1920 illustrated in FIG. 18D to bedisplayed.

The browser window 1895 may be used to display information of aresult(s) indicated by information in the display window 1890. In atleast one embodiment, selecting the URL text displayed in the displaywindow 1890 may cause a web page associated with the URL to be renderedin the browser window 1895. Information regarding a result(s) may bedisplayed in tabs within the browser window and the browser window andtabs may be scaled and otherwise manipulated as is well known in theexisting art. This may allow a guide or a user of the GUI 1850 to makean examination of a result(s).

As illustrated in FIG. 18C, an exemplary GUI 1900 is shown forrecommending that a search result(s) may be associated with a queryand/or a profile and/or a categorization within a taxonomy, etc. The GUI1900 includes selection controls 1905 and action controls 1910.

The selection controls 1905 may be used to select and/or modify thecontent of and/or information associated with a query. This may alterthe association of ratings data associated with a result, may create anew request or query record, etc. In the example illustrated in FIG.18C, the ‘primary phrase’ selection box 1905 a may be used to alter theprimary phrase or query phrase (which may be stored in a query record)in order to form a different query. The selection control 1905 a may beimplemented using a drop-down list, a prompted typing box, or otherinterface. The category selection control 1905 b may be used to select acategorization within a taxonomy. The subcategory selection control 1905c may be used to select a subcategory to be associated with a query. Inother words, a branch within a taxonomy may be selected using theselection controls 1905. In at least one embodiment, the filteringoptions may be more or less complex than those illustrated as desired toeffectively operate the embodiments.

The action controls 1910 may be used to take various actions. In theexample illustrated in FIG. 18C, the ‘more’ button 1910 a may enable amore complex filtering GUI which may be used to make modifications to aquery and/or information associated with the query. The ‘add newrecommendation’ button 1910 b may be used to select a result to beassociated with a query and information selected. In at least oneembodiment, the GUI 1940 illustrated in FIG. 18E may be provided whenthe ‘add new recommendation’ button 1910 b is activated. The ‘cancel’button 1910 c may be used to close the GUI 1900 and return to the GUI1850.

As illustrated in FIG. 18D, an exemplary GUI 1920 is shown forrecommending a category and/or other information to be associated with aquery. The GUI 1920 includes selection controls 1925 and action controls1930.

The selection controls 1925 may be used to select and/or modify thecontent of and/or information associated with a query or search request.This may alter the association of ratings data associated with a result,may create a new request or search query, etc. In the exampleillustrated in FIG. 18D, the ‘primary phrase’ selection control 1925 amay be used to alter the primary phrase (which may be stored in a queryrecord) in order to form a query. This may be done using a drop-downlist, a prompted typing box, or other interface. The ‘category’selection control 1925 b may be used to select a root term within ataxonomy. The ‘subcategory’ selection control 1925 c may be used toselect a topic. In other words, a branch within the taxonomy (one typeof profile) may be selected using controls such as the selectioncontrols 1925. In at least one embodiment, the filtering options may bemore or less complex than those illustrated as required to effectivelyoperate the embodiments.

The action buttons 1930 may be used to take actions. In the exampleillustrated in FIG. 18D, the ‘more’ button 1930 a leads to a morecomplex filtering GUI which may allow a user to make modifications tothe query text and/or information associated with a query. The ‘save newcategory’ button 1930 b may be used to record information selected (e.g.a categorization of a query) and return to the GUI 1850. The ‘cancel’button 1930 c may be used to close the GUI 1920 and return to the GUI1850 without making modifications to information associated with aquery.

As illustrated in FIG. 18E, an exemplary GUI 1940 is shown forassociating a result with a query. The GUI 1940 includes a scorecardindicator 1820, a query field 1855, a return button 1945, a browserwindow 1950, selection tabs 1955, a new browser button 1957, a recommendbutton 1960, a URL entry box 1962, locator controls 1965, a searchtarget window 1967, a categorization indicator 1970 and a searchmodification control 1975.

The functionality of the scorecard indicator 1820 and the query field1855 were previously explained with respect to FIG. 18B. The returnbutton 1945 may be used to return to the voting GUI 1850 withoutrecommending a new result to be associated with the selected queryand/or profile and/or categorization.

The browser window 1950 may be used to locate and select a result. Inthe example illustrated in FIG. 18E, the URL entry 1962 is providedwhich may be used to enter the target URL of a resource. An additionalbrowser window(s) and/or tabs may be opened using the ‘new browser’button 1957. Multiple browser tabs may be opened, as illustrated in FIG.18E, and a user may navigate between them using controls well known inthe relevant art.

The selection tabs 1955 may be used to view a browser window such as thebrowser window 1950 for obtaining a search result, and/or other GUIssuch as a resource management GUI described in the related U.S.application Ser. No. 11/774,852 previously mentioned. The locatorcontrols 1965 may be used to locate occurrences of an item in a web pageor other resource rendered in a browser window, such as the browserwindow 1950. The items may include but are not limited to elements of aquery. A query may include text and/or other media. Such information maybe displayed in the query field 1855. In the example illustrated in FIG.18E, the ‘find previous’ control 1965 a and the ‘find next’ control 1965b may be used to navigate within the content of a web page and locateinformation which may provide a result relating to a query and/orinformation selected. Information to be located in the resource renderedin the browser window 1950 may be displayed in the search target window1967. The search target window 1967 may be used to identify informationto be located in a resource. For example, any or all of query may beentered in the search target window 1967.

The recommend button 1960 may be used to indicate completion of the taskof locating a search result and a desire to associate a result with aselected query and/or other information. In at least one embodiment, therecommend button 1960 may be inactive based on information indicated inthe search database 120 (FIG. 1). For example the ‘recommend result’button 1960 may be inactive if a guide does not have sufficient pointsto allow a recommendation to be performed.

The categorization indicator 1970 may indicate a category and/or otherinformation associated with a query contained in the query field 1855.For example, the categorization indicator 1970 might contain informationregarding a category and/or a profile, etc. associated with a query.

The search modification control 1975 may be used to modify a query, acategorization and/or other information associated with a result(s). Inthe example illustrated in FIG. 18E, the ‘modify search’ button 1975 isprovided which may cause a GUI such as GUI 1920 (FIG. 18D) to bepresented.

As illustrated in FIG. 18F, an exemplary GUI 1980 is shown for reviewinga category associated with a query. The GUI 1980 includes filtercontrols 1805, scorecard indicator 1820, close button 1825, availabilityselection control 1830, logoff control 1835, selection tabs 1840, votingcontrols 1843, query categorization review window 1845, navigationcontrols 1848 and ‘submit votes’ button 1847. The function of thescorecard indicator 1820, the close button 1825, the availabilityselection control 1830, the logoff control 1835, and selection tabs 1840were previously explained with respect to FIG. 18A. The filter controls1805 may include elements which may be used to designate a filter(s)which may be applied to a search request(s) and/or other items. Usingthe example illustrated in FIG. 18F, the ‘pending only’ filter control1805 h may be used to select all search queries or only search querieswhich are pending voting activities by activating and/or deactivatingthe check box. The category filter control 1805 j may be used to selectone or more root terms which may be used to select a search request. Thesubcategory filter control 1805 k may be used to select one or moretopics which may be used to select a search request. The ‘remove filter’control 1805 m may be used to clear all filter controls selected usingthe GUI 1980. The ‘apply filter’ control 1805 n may be used to apply afilter condition selected using the GUI 1980.

The query categorization review window 1845 displays information relatedto a query(ies) and a category(ies) or other information associated witha query, and allows a user of the GUI 1980 to make indications ofhis/her opinion(s). In the embodiment illustrated in FIG. 18F, anindicator of the status of a query, such as a colored flag may beprovided to identify queries that are pending voting by a guide(s) maybe provided. Query text (e.g., ‘.net framework’) is displayed; acategory indicator, a subcategory indicator, and voting controls 1843are provided. The voting controls 1843 may be used to indicate judgmentof the relevance of a category and/or subcategory assigned to a query.Options may include ‘good’, ‘bad’, ‘don't know’, and ‘remove’ which mayindicate a positive, negative, neutral, or ‘veto’ opinion, respectively.Queries for which a user of the GUI 1980 may not vote, based uponinformation indicated in the search system database 120 (FIG. 1) may beindicated by being ‘grayed out’ as shown in FIG. 18F.

The ‘submit votes’ button 1847 may be used to indicate the intent torecord votes or opinions obtained using the GUI 1980. Activation of the‘submit votes’ button 1847 may cause the search system 130 (FIG. 1) toupdate information contained in the search database 120 to reflectinformation obtained using the GUI 1980 (FIG. 18F). The navigationcontrols 1848 may be used to navigate within the query categorizationreview window 1845.

As previously described herein, a comparison voting method may be usedto determine a ‘best’ result by comparing at least two search results.It may be desirable that a group(s) of results under consideration maybe displayed to a person who is making a comparison. A side-by-sidepresentation may improve a person's ability to compare the content of agroup(s) of result(s).

As illustrated in FIG. 18G, an exemplary GUI 2000 is provided forcomparison voting regarding a group of result(s) associated with aquery. The GUI 2000 includes scorecard indicator 1820, query field 1855,categorization indicator 1970, search modification control 1975, firstresult window 2005, second result window 2010, result identifiers 2012,preference indication controls 2015, and action buttons 2020.

The scorecard indicator 1820 and the query field 1855 were previouslydescribed with respect to FIG. 18B. The categorization indicator 1970and search modification control 1975 were previously described withrespect to FIG. 18E.

The first result window 2005 may be used to view information associatedwith a first result which has been selected for comparison. The secondresult window 2010 may be used to view information associated with asecond result which has been selected for comparison. The GUI 2000 maycontain any number of result windows. Result windows are displayedsimultaneously to a user of the GUI 2000. The result identifiers 2012may be used to indicate which content is associated with an option(s)indicated in the preference indication controls 2015. Using the exampleillustrated in FIG. 18G, the result identifier 2012 a may indicate thatthe first result window 2005 is rendering option ‘A’, and the resultidentifier 2012 b may indicate that the second result window 2010 isrendering option ‘B’.

The preference indication controls 2015 may be used to indicate whichsearch result is judged superior. Using the example illustrated in FIG.18G, checkboxes are provided to select result ‘A’ or ‘B’. In at leastone embodiment, only one result may be selected as the preferred result.As previously described herein, two or more items may be presented forconsideration using the GUI 2000.

The action buttons 2020 may be used to take actions relating topreferences indicated using the GUI 2000. Using the example illustratedin FIG. 18G, the ‘submit votes’ button 2020 a may be used to indicatethe intent to record preferences in the search system database 120 (FIG.1). The ‘cancel’ button 2020 b may be used to indicate the intent toexit the GUI 2000 and discard information obtained using the GUI 2000(FIG. 18G).

While the embodiments herein have been described with respect to theGUI's depicted in FIGS. 18A-18G, other interfaces, and alternate waysfor indicating actions known within the relevant art may be employedwithout departing from the spirit of the embodiments described herein.For example, a voice controlled interface, (e.g., voice xML) amotion-based input (e.g., gyroscopic or other mechanically based motiondetection), a keypad based input (e.g., a phone keypad), etc. may beutilized to provide a suitable interface(s) to operate the embodiments.

While the GUIs illustrated in FIGS. 18A-18G have been illustrated usingthe example of reviewing a result(s) associated with a search query orsearch request, no limitation is implied thereby. Any review item(s)associated with a reference item(s) may be selected and/or reviewedusing the GUIs illustrated herein. For example, a search resource(s)associated with a category and/or other information might be selectedand reviewed using the tools described herein, or an advertisement(s)associate with a profile and/or other information might be selected andreviewed.

As illustrated in FIG. 19, an exemplary GUI is provided for selection ofa game associated with a category, topic, word, phrase, or anycombination thereof. The GUI 2100 includes an availability selectioncontrol 2105, a logoff control 2110, a game information indicator 2115,a activity selection tabs 2120, a search button 2125, a search box 2126,game location tabs 2130, an advertisement window 2135, game informationindicators 2140, a recent game display 2145, a recent game selectioncontrols 2147, a scoreboard indicator 2150, action buttons 2155, gameselection controls 2160, a game information window 2165, and navigationcontrols 2170. The GUI 2100 may be used to select a game(s) which may beplayed as a solo activity. Games may be selected using filteringcriteria. For example, games may be selected by entering and/orselecting a topic, keyword, category, etc or based on a skill level, orany other information which may be associated with a game. For example,games might be associated with an affiliate group, a profile, etc. Aprofile might include information such as a time interval, geographiclocation, personality characteristics, or any other item(s). Selectionsavailable may be based on information associated with a person(s). Forexample, some games may be available only to persons associated with anitem, or with a set of skills, etc.

The availability selection control 2105 may be used to indicate astatus. In at least one embodiment the availability selection control2105 may be used to indicate availability for various activities. Asshown in FIG. 19, a drop-down menu may be provided from which a list ofavailability options may be selected. Any options and/or selectiontechniques might be employed to select a status within the spirit ofthis disclosure. For example, activation of the availability selectioncontrol 2105 may provide a menu of activities, or might be as simple asa toggle of a check-box.

The logoff controls 2110 may be used to log off of the search system 130(FIG. 1) and may include navigation controls which may be used to selectother GUI's, etc.

The system information indicator 2115 may include information relatingto aggregated information of players and/or games etc. which are inprogress, or historical information, or any information regarding systemstatus.

The GUI 2100 may include one or more activity selection tabs such asselection tabs 2120. The activity selection tabs 2120 may be used toselect activities, and/or to review other items. Using the exampleillustrated in FIG. 19, the ‘Solo Game’ tab 2120 b may cause the GUI2100 to be displayed, the ‘Group Game’ tab 2120 a may cause a GUI suchas GUI the 2300 illustrated in FIG. 21 to be presented, the ‘My Account’tab 2120 c, the ‘My Points’ tab 2120 d, and the ‘My Profile tab 2120 emay display other GUIs which may be used for review of information.

The search box 2126 may be used to enter a search term. The searchbutton 2125 may be used to initiate a search based on the content of thesearch box 2126. Activation of the search button 2125 may cause the GUI2200 (FIG. 22) to be presented. Alternately, entry of a term in thesearch box 2126 may cause games associated with the content of thesearch box to be displayed in the game information window 2165

The game location tabs 2130 may be used to view GUIs such as the GUI2100. The selection tab 2130 a (‘Pick a Game’) may present table formatfor selection of a game, and the selection tab 2130 b (‘Search results’)may present a GUI such as the GUI 2500 illustrated in FIG. 23.

The advertisement window 2135 may be used to present an advertisementand/or other information. Content of the advertising window 2135 may bemodified based on other information selected in the GUI 2100. Forexample, a category or keywords, etc. selected may cause informationpresented in the advertising window 2135 to be modified. More than oneadvertisement window 2135 may be included in a GUI such as the GUI 2100,the GUI 2200, the GUI 2250, the GUI 2275, the GUI 2300 and the GUI 2400.

The game information indicators 2140 may present information regardingtypes of information associated with a game(s). The ‘Game Topic’ gameinformation indicator 2140 a may indicate a category. A left-click onthe ‘Game Topic’ game information indicator 2140 a may sort games in thegame information window 2165 by categories associated with the game. Aright-click on the ‘Game Topic’ game information indicator 2140 a maypresent a drop-down list of categories. The ‘Skill Level’ gameinformation indicator 2140 b may indicate a skill. A left-click on the‘Skill Level’ game information indicator 2140 b may sort games in thegame information window 2165 by skill level associated with the game. Aright-click on the ‘Skill Level’ game information indicator 2140 b maypresent a drop-down list of skill levels. Any number of game informationindicators 2140 may be provided in the GUI 2100, and it may be possibleto add and/or remove game information indicators 2140. For example a‘type’ indicator (comparative, good/bad, comparison, etc.) might beprovided as a game information indicator.

The recent game display 2145 may be used to indicate information ofgames which have been played recently. Any type of informationassociated with a game may be presented in the recent game display 2145.The recent game selection controls 2147 may indicate recently playedgames, and may be activated to provide information regarding games. Forexample the ‘+’ indicator in the game selection indicators might expandthe information displayed in the recent game display 2145. Activation ofa recent game selection control may cause games with similarcharacteristics to be displayed in the game information window 2165.Activation of the ‘Sports’ game selection control 2147 a may take a userdirectly to a game related to the topic ‘Sports’. Activation of the‘Entertainment>Movies’ game selection control 2147 b may take a userdirectly to a game related to the topic ‘Entertainment>Movies’.Activation of the ‘Sports>Baseball’ game selection control 2147 c maytake a user directly to a game related to the topic ‘Sports>Baseball’.Activation of the ‘Manage Your Games’ game selection control 2147 d maycause a GUI for editing and/or reviewing a game usage history to beprovided.

The scoreboard indicator 2150 provides information of ‘points’. Any typeof scoring indications may be provided. The ‘current’ scoreboardindicator 2150 a may indicate points accrued during a game session. The‘lifetime’ scoreboard indicator 2150 b may indicate total pointsaccrued. The ‘Level’ scoreboard indicator 2150 c may indicate a skilllevel. For example, a skill level associated with a user of the GUIwhich may be related to a topic(s), etc. may be indicated in the ‘Level’scoreboard indicator 2150 c.

The action buttons 2155 may be used to take actions regardinginformation indicated in the GUI 2100. The ‘Go to Game’ action button2155 may be used to activate a game GUI such as the GUI's 2200, 2250,and 2275 illustrated in FIGS. 20A-20C associated with a selected gameselection control 2160. For example if the game selection indicator 2160b is selected, activation of the ‘Go to Game’ action button 2155 mightpresent a GUI such as the GUI 2200 associated with the category‘Sports>Baseball’ at the ‘Expert’ level of skill.

The game selection controls 2160 may be used to select a game associatedwith a game selection control 2160. For example, the game selectioncontrol 2160 a may be used to select the ‘Novice’ level game associatedwith ‘Sports>Football’, a ‘left’ click might select the game selectionindicator 2160 a and a ‘double’ click might cause a GUI such as the GUI2200 (FIG. 20A) associated with the ‘Sports>Football’ category at the‘Novice’ skill level to be presented. Likewise, the game selectionindicator 2160 c may be used to select the ‘Novice’ level gameassociated with the category ‘Entertainment>Dining>Indianapolis’, aleft-click might select the game selection indicator 2160 c and a doubleclick might cause a GUI such as the GUI 2200 (FIG. 20A) associated withthe ‘Entertainment>Dining>Indianapolis’ category at the ‘Novice’ skilllevel to be presented. The navigation controls 2170 may be used tonavigate within the game information window 2165.

As illustrated in FIG. 20A, an exemplary GUI is provided for votingregarding a review item associated with reference item. The GUI 2200 maybe presented as a single panel such as the voting window 2220, or may bean element of a GUI such as the GUI 2400 illustrated in FIG. 22. The GUI2200 may include availability selection control 2105, logoff control2110, advertisement window 2135, scoreboard indicator 2205, system topicinformation indicator 2210, activity selection tabs 2215, votinginformation window 2220, voting controls 2225, reference item indicator2230, review item indicator 2235, and action controls 2240. The GUI 2200may be used to indicate opinion regarding a review item related to areference item.

The function of the availability selection control 2105, logoff control2110 and the advertisement window 2135 were previously explained withrespect to FIG. 19.

The scoreboard indicator 2205 provides information of scoring. Any typeof scoring indications may be provided. The scoreboard indicator 2205 amay indicate points accrued by a player during a game session. Thescoreboard indicator 2150 b may indicate the highest point total accruedin a game. The scoreboard indicator 2150 c may indicate a scoreassociated with a different player. A scoreboard indicator 2205 mayindicate any type of information relating to scores, and it may bepossible to add or remove scoreboard indicators in the GUI 2200.

The system topic information indicator 2210 may include informationrelating to aggregated information of players and/or games etc. whichare in progress, or historical information, or any information regardingsystem status. For example, the system topic information indicator 2210may indicate system information which is related to a game topic relatedto the voting in progress.

The selection tabs 2215 may allow selection of a GUI, and/or mayindicate an activity. The selection tab 2215 a may be used to select‘Games’ as an activity and/or GUI. The selection tab 2215 b may be usedto indicate that the activity ‘Search’ has been selected. Any number ofselection tabs 2215 may be presented in the GUI 2200.

The voting information window 2220 may be used to present informationassociated with voting. Any type of information might be presented inthe voting information window 2220. For example, images, video, or othertypes of media might be present in the voting information window 2220.

The reference item indicator 2230 may include information associatedwith a reference item. While a text display is used as the referenceitem indicator 2230 for the purposes of illustration in FIG. 20A, anytype of media and/or information associated with a reference item mightbe presented. For example, an image(s), an audio clip, a hyperlinkand/or other types of information associated with a reference item mightbe included in the reference item indicator 2230.

The review item indicator 2235 may include information associated with areview item. While a text display is used as the review item indicator2235 for the purposes of illustration in FIG. 20A, any type of mediaand/or information associated with a review item might be presented. Forexample, an image(s), an audio clip, a hyperlink and/or other types ofinformation associated with a review item might be included in thereview item indicator 2235.

The voting controls 2225 may be used to indicate an opinion regarding areview item. The voting controls 2225 are mutually exclusive. A user mayselect exactly one voting control to register an opinion regarding eachreview item indicated in the review item information indicator 2235. Thevoting control 2225 a allows a user to register a ‘Wow’ or positiveopinion regarding a review item indicated in the review item indicator2235. The voting control 2225 b allows a user to register a ‘Not’ ornegative opinion regarding a review item indicated in the review itemindicator 2235. The voting control 2225 c allows a user to register an‘I don't know’ or neutral opinion regarding a review item indicated inthe review item indicator 2235. A vote may be indicated by for exampleclicking on a voting control which may cause an indicator such as afilled dot such as that indicated in the voting control 2225 b to bedisplayed.

Any number of review item information indicators 2235 and votingcontrols 2225 associated with a review item may be presented in thevoting information window 2220 in the GUI 2200.

The action controls 2240 may be used to take actions regardinginformation obtained using the GUI 2200. The ‘Vote’ action control 2240a may be used to indicate that opinions indicated in the GUI 2200 are tobe recorded. The ‘Skip’ action control 2240 b may be used to indicatethat a review item is to be skipped without voting. The ‘Exit’ actioncontrol 2240 c may be used to indicate that a game session or votingsession is to be terminated.

As illustrated in FIG. 20B, an exemplary GUI is provided for votingregarding group of review items associated with reference item. The GUI2250 may be presented as a single panel such as the voting window 2220,or may be an element of a GUI such as the GUI 2400 illustrated in FIG.22. The GUI 2200 may include availability selection control 2105, logoffcontrol 2110, advertisement window 2135, scoreboard indicator 2205,system topic information indicator 2210, activity selection tabs 2215,voting information window 2220, voting controls 2265, reference itemindicator 2255, review item indicator 2260, and action controls 2240.The GUI 2250 may be used to indicate opinion regarding review itemsrelated to a reference item.

The function of the availability selection control 2105, logoff control2110 and the advertisement window 2135 were previously explained withrespect to FIG. 19. The function of the scoreboard indicator 2205,system topic information indicator 2210, activity selection tabs 2215,voting information window 2220, and the action controls 2240 werepreviously explained with respect to FIG. 20A.

The reference item information indicator 2255 may include informationassociated with a reference item. While a text display is used as thereference item indicator 2225 for the purposes of illustration in FIG.20B, any type of media and/or information associated with a referenceitem might be presented. For example, an image(s), an audio clip, ahyperlink and/or other types of information associated with a referenceitem might be included in the reference item indicator 2255.

The review item information indicators 2260 may include informationassociated with review items. While a text display is used as a reviewitem indicator 2260 for the purposes of illustration in FIG. 20B, anytype of media and/or information associated with a review item might bepresented. For example, an image(s), an audio clip, a hyperlink and/orother types of information associated with a review item might beincluded in the review item indicator 2260. At least two review itemindicators are provided in the GUI 2250. In a preferred embodiment,exactly two review item indicators are presented. The review itemindicator 2260 a may present information associated with a first reviewitem such as the answer ‘A George Washington, C in C Continental Army’.The review item indicator 2260 b may present information associated witha second review item such as the answer ‘B George Washington, Martha'shusband’.

The voting controls 2265 may be used to indicate an opinion regarding areview item. The voting controls 2265 are mutually exclusive. A user mayselect exactly one voting control to register an opinion regarding eachreview item indicated in the review item information indicator 2255. Thevoting control 2265 a allows a user to register a positive opinionregarding a review item indicated in the review item indicator 2260 a.The voting control 2265 b allows a user to register a positive opinionregarding a review item indicated in the review item indicator 2260 b. Avote may be indicated by for example clicking on a voting control whichmay cause an indicator such as a filled dot such as that indicated inthe voting control 2265 a to be displayed.

Any number of groups of review item indicators 2260 and mutuallyexclusive voting controls 2265 may be presented along with an associatedreference item indicator 2255 in the voting information window 2220 inthe GUI 2250.

As illustrated in FIG. 20C, an exemplary GUI is provided for votingregarding group of review items associated with reference item. The GUI2275 may be presented as a single panel such as the voting window 2220,or may be an element of a GUI such as the GUI 2400 illustrated in FIG.22. The GUI 2200 may include availability selection control 2105, logoffcontrol 2110, advertisement window 2135, scoreboard indicator 2205,system topic information indicator 2210, activity selection tabs 2215,voting information window 2220, voting controls 2290, reference itemindicator 2280, review item indicator 2285, and action controls 2240.The GUI 2275 may be used to indicate opinion regarding review itemsrelated to a reference item.

The function of the availability selection control 2105, logoff control2110 and the advertisement window 2135 were previously explained withrespect to FIG. 19. The function of the scoreboard indicator 2205,system topic information indicator 2210, activity selection tabs 2215,voting information window 2220, and the action controls 2240 werepreviously explained with respect to FIG. 20A.

The reference item information indicator 2280 may include informationassociated with a reference item. While a text display is used as thereference item indicator 2280 for the purposes of illustration in FIG.20B, any type of media and/or information associated with a referenceitem might be presented. For example, an image(s), an audio clip(s), ahyperlink(s) and/or other types of information associated with areference item might be included in the reference item indicator 2280.

The review item information indicators 2285 may include informationassociated with review items. While a text display is used as a reviewitem indicator 2285 for the purposes of illustration in FIG. 20B, anytype of media and/or information associated with a review item might bepresented. For example, an image(s), an audio clip(s), a hyperlink(s)and/or other types of information associated with a review item might beincluded in the review item indicator 2285. At least two review itemindicators are provided in the GUI 2275. In a preferred embodiment, atleast three review item indicators are presented. The review itemindicator 2285 a may present information associated with a first reviewitem such as the answer ‘A Thomas Jefferson’. The review item indicator2285 b may present information associated with a second review item suchas the answer ‘B King George, III’. The review item indicator 2285 c maypresent information associated with a third review item such as theanswer ‘C George Washington’. The review item indicator 2285 d maypresent information associated with a fourth review item such as theanswer ‘D George H. Bush’.

The voting controls 2290 may be used to indicate an opinion regarding areview item. The voting controls 2290 may be mutually exclusive (i.e.,only one review item may be give a rating). A user may select exactlyone ranking in each voting control 2290 to register an opinion regardingeach review item indicated in the review item information indicator 2285associated with the review item and the voting control 2290. The votingcontrol 2290 a allows a user to indicate an opinion regarding a rankingof a review item indicated in the review item indicator 2285 a. Thevoting control 2290 b allows a user to indicate an opinion regarding aranking of a review item indicated in the review item indicator 2285 b.The voting control 2290 c allows a user to indicate an opinion regardinga ranking of a review item indicated in the review item indicator 2285c. The voting control 2290 d allows a user to indicate an opinionregarding a ranking of a review item indicated in the review itemindicator 2285 d. The number of positions in the voting controls 2290may be the same as the number of reference item indicators 2285.

A vote may be indicated by for example clicking on a voting controlwhich may cause an indicator such as a filled dot to be displayed. Forexample, the dot in the position ‘4’ as indicated in the voting control2290 a, the dot in the position ‘3’ as indicated in the voting control2290 b, the dot in the position ‘2’ as indicated in the voting control2290 c, and the dot in the position ‘1’ as indicated in the votingcontrol 2290 d.

Any number of groups of review item indicators 2285 and controls 2290associated with a review item indicator may be presented along with anassociated reference item indicator 2280 in the voting informationwindow 2220 in the GUI 2275.

Alternately, it may be possible to select any position in a votingcontrol 2290 regardless of a vote which has been indicated in othervoting controls. The voting controls 2290 have been illustrated using‘radio button’ indicators, but other interfaces such as check boxes,slider bars, etc. which is well known in the art might be utilized.

As illustrated in FIG. 21, an exemplary GUI is provided for selection ofa multi-player game associated with a category, topic, word, phrase, orany combination thereof. The GUI 2300 includes availability selectioncontrol 2105, logoff control 2110, game information indicator 2115,activity selection tabs 2120, search button 2325, search box 2326, gamelocation tabs 2330, advertisement window 2135, game informationindicators 2340, recent game display 2345, recent game selectioncontrols 2347, scoreboard indicator 2150, action buttons 2355, gameselection controls 2360, game information window 2365, and navigationcontrols 2370. The GUI 2300 may be used to select a game(s) which may beplayed as a multi-player activity. Games may be selected using filteringcriteria. For example, games may be selected by entering and/orselecting a topic, keyword, category, etc or based on a skill level, orany other information which may be associated with a game. For example,games might be associated with an affiliate group, a profile, etc. Aprofile might include information such as a time interval, geographiclocation, personality characteristics, or any other item(s). Selectionsavailable may be based on information associated with a person(s). Forexample, some games may be available only to persons associated with anitem, or with a set of skills, etc.

The function of the availability selection control 2105, logoff control2110 game information indicator 2115, activity selection tabs 2120,scoreboard indicator 2150, and the advertisement window 2135 werepreviously explained with respect to FIG. 19.

The game location tabs 2330 may be used to view GUIs such as the GUI2300. The selection tab 2330 a (‘Pick a Game’) may present table formatfor selection of a game, and the selection tab 2330 b (‘Search results’)may present a GUI such as the GUI 2500 illustrated in FIG. 23.

The advertisement window 2135 may be used to present an advertisementand/or other information. Content of the advertising window 2135 may bemodified based on other information selected in the GUI 2300. Forexample, a category or keywords, etc. selected may cause informationpresented in the advertising window 2135 to be modified. More than oneadvertisement window 2135 may be included in the GUI 2300.

The game information indicators 2340 may present information regardingtypes of information associated with a game(s). The ‘Game Topic’ gameinformation indicator 2340 a may indicate a category. A left-click onthe ‘Game Topic’ game information indicator 2340 a may sort games in thegame information window 2365 by categories associated with the game. Aright-click on the ‘Game Topic’ game information indicator 2340 a maypresent a drop-down list of categories. The ‘Skill Level’ gameinformation indicator 2340 b may indicate a skill. A left-click on the‘Skill Level’ game information indicator 2340 b may sort games in thegame information window 2365 by skill level associated with the game. Aright-click on the ‘Skill Level’ game information indicator 2340 b maypresent a drop-down list of skill levels. The ‘Players’ game informationindicator 2340 c may indicate a number of participants in a game. Aleft-click on the ‘Players’ game information indicator 2340 c may sortgames in the game information window 2365 by number of participantsassociated with the game. A right-click on the ‘Players’ gameinformation indicator 2340 c may present a drop-down list. Any number ofgame information indicators 2340 may be provided in the GUI 2300, and itmay be possible to add and/or remove game information indicators 2340.For example a ‘type’ indicator (comparative, good/bad, comparison, etc.)might be provided as a game information indicator 2340.

The recent game display 2345 may be used to indicate information ofgames which have been played recently. Any type of informationassociated with a game may be presented in the recent game display 2345.The recent game selection controls 2347 may indicate recently playedgames, and may be activated to provide information regarding games. Forexample the ‘+’ indicator in the game selection indicators might expandthe information displayed in the recent game display 2345. Activation ofa recent game selection control may cause games with similarcharacteristics to be displayed in the game information window 2365.Activation of the ‘Sports’ game selection control 2347 a may take a userdirectly to a game related to the topic ‘Sports’. Activation of the‘Entertainment>Movies’ game selection control 2347 b may take a userdirectly to a game related to the topic ‘Entertainment>Movies’.Activation of the ‘Sports>Baseball’ game selection control 2347 c maytake a user directly to a game related to the topic ‘Sports>Baseball’.Activation of the ‘Manage Your Games’ game selection control 2347 d maycause a GUI for editing and/or reviewing a game usage history to beprovided.

The action buttons 2355 may be used to take actions regardinginformation indicated in the GUI 2300. The ‘Go to Game’ action button2355 a may be used to activate a game GUI such as the GUI 2400illustrated in FIG. 22 associated with a selected game selection control2360. For example if the game selection indicator 2360 b is selected,activation of the ‘Go to Game’ action button 2355 a might present a GUIsuch as the GUI 2400 (FIG. 22) associated with the category‘Sports>Baseball’ at the ‘Expert’ level of skill. The action button 2355b may be used to view information of players associated with a gameselection indicator 2360. For example, if the game selection indicator2360 c is selected, activation of the action button 2355 b may cause alist of players associated with the game indicated in the game selectionindicator 2360 c to be displayed. Any number of action buttons 2355 maybe included in the GUI 2300.

The game selection controls 2360 may be used to select a game associatedwith a game selection control 2360. For example, the game selectioncontrol 2360 a may be used to select the ‘Novice’ level game associatedwith ‘Sports>Football’, a ‘left’ click might select the game selectionindicator 2360 a and a ‘double’ click might cause a GUI such as the GUI2400 (FIG. 21) associated with the ‘Sports>Football’ category at the‘Novice’ skill level to be presented. Likewise, the game selectionindicator 2360 c may be used to select the ‘Novice’ level gameassociated with the category ‘Entertainment>Dining>Indianapolis’, aleft-click might select the game selection indicator 2360 c and a doubleclick might cause a GUI such as the GUI 2400 (FIG. 24) associated withthe “Entertainment>Dining>Indianapolis’ category at the ‘Novice’ skilllevel to be presented. The navigation controls 2370 may be used tonavigate within the game information window 2365.

As illustrated in FIG. 22, an exemplary GUI is provided forparticipation in a multi player game is provided. The GUI 2400 includesan availability selection control 2105, a logoff control 2110, anactivity selection tabs 2120, an advertisement window 2135, a scoreboardindicator 2150, a game information indicator 2405, a player indicator2410, a player selection indicator 2415, a chat window 2420, playervoting window 2445, a reference item indicator 2425, voting controls2435, a review item indicator 2440, action controls 2430, wageringcontrols 2450, and a mode selector 2455. The GUI 2400 may be used toparticipate in a game(s) which may be played as a multi-player activity.

The function of the availability selection control 2105, logoff control2110, activity selection tabs 2120, scoreboard indicator 2150, and theadvertisement window 2135 were previously explained with respect to FIG.19.

The game information indicator 2405 may be used to present informationassociated with a game. For example, the topic of a game, timeinformation, usage information, or any other information related toplayers, games, or other status may be presented in the game informationindicator 2405.

The player indicator 2410 may indicate information associated withplayers participating in a multi-player game. For example a ‘handle’associated with a user, an image or avatar, a skill level, a number ofpoints, etc. might be presented in a player indicator 2410. The playerindicator 2410 a may indicate the player ‘JessB’ at the ‘Novice’ skilllevel. The player indicator 2410 b may indicate the player ‘BobH’ at the‘Expert’ skill level. The player indicator 2410 c may indicate theplayer ‘JaneS’ at the ‘Novice’ skill level. The player indicator 2410 dmay indicate the player ‘JoanJ’ at the ‘Journeyman’ skill level. Theplayer indicator 2410 e may indicate the player ‘GeneO’ at the ‘Expert’skill level. The player indicator 2410 f and 2410 g may indicate that avacant space exists and that new players may join the game.

The player selection indicator 2415 may indicate information associatedwith a selection made by players participating in a multi-player game.For example a letter indicated in a review item indicator 2440 may bedisplayed once all players have registered an opinion using a controlsuch as the action buttons 2430. The player selection indicator 2415 amay indicate the player ‘JessB’ selected review item A. The playerselection indicator 2415 b may indicate the player ‘BobH’ selectedreview item B. The player selection indicator 2415 c may indicate theplayer ‘JaneS’ selected review item B. The player selection indicator2415 d may indicate the player ‘JoanJ’ selected review item A. Theplayer selection indicator 2415 e may indicate the player ‘GeneO’selected review item B. Automated responses based on information storedin and/or derived from the database 120 (FIG. 1) may be presented usinga player indicator and a player selection indicator. For example,responses associated with a solo session may be presented using a playerindicator 2410 and a player selection indicator 2415.

The chat window 2420 may be used to allow players participating in amulti-player game to exchange messages via a ‘chat’ function.

The player voting window 2445 may display a GUI such as the GUI 2200(FIG. 20A), 2250 (FIG. 20B), 2275 (FIG. 20C) or any other voting GUI.Information presented in the player voting window 2445 may be presentedto any or all players.

The reference item indicator 2425 may include information associatedwith a reference item. While a text display is used as the referenceitem indicator 2425 for the purposes of illustration in FIG. 22, anytype of media and/or information associated with a reference item mightbe presented. For example, an image(s), an audio clip(s), a hyperlink(s)and/or other types of information associated with a reference item mightbe included in the reference item indicator 2425. The reference itemindicator 2425 indicates that the query ‘Who was the first president ofthe United States?’ is the reference item presented in the player votingwindow 2445.

The review item indicator 2440 may include information associated with areview item. While a text display is used as the review item indicator2440 for the purposes of illustration in FIG. 22, any type of mediaand/or information associated with a review item might be presented. Forexample, an image(s), an audio clip(s), a hyperlink(s) and/or othertypes of information associated with a review item might be included inthe review item indicator 2440. The review item indicator 2440 aindicates that ‘A George Washington’ is associated with a first reviewitem. The review item indicator 2440 b indicates that ‘B GeorgeWashington, Martha's husband’ is associated with a second review item.Any or all information indicated in a review item indicator may bepresented in a player selection indicator 2415 to indicate a selectionmade.

The voting controls 2435 may be used to indicate an opinion regarding areview item. Using the example in FIG. 22, the voting controls 2435 aremutually exclusive. A user may select exactly one voting control toregister an opinion regarding each review item indicated in the reviewitem information indicator 2425. The voting control 2435 a allows a userto register a positive opinion regarding a review item indicated in thereview item indicator 2440 a. The voting control 2435 b allows a user toregister a positive opinion regarding a review item indicated in thereview item indicator 2440 b. A vote may be indicated by for exampleclicking on a voting control which may cause an indicator such as afilled dot such as that indicated in the voting control 2435 b to bedisplayed. While a particular example of a voting GUI has been used forthe purposes of illustration in FIG. 22, no limitation is impliedthereby.

The action controls 2430 may be used to take actions regardinginformation obtained using the GUI 2400. The ‘Vote’ action control 2430a may be used to indicate that opinions indicated in the GUI 2400 are tobe recorded. The ‘Skip’ action control 2430 b may be used to indicatethat a review item is to be skipped without voting. The ‘Exit’ actioncontrol 2430 c may be used to indicate that a game session or votingsession is to be terminated. Activation of the action control 2430 c mayclose the GUI 2400 and return to a GUI such as the GUI 2300 (FIG. 21).

The wagering controls 2450 may be used to modify a number of pointswhich may be awarded to and/or lost by a player based on the outcome ofa game. For example, a player might be allowed to increase or decrease anumber of points wagered on an outcome, such as whether the player willagree with other players, will agree with an expert, and any other typeof outcome. The wagering control 2450 a, the ‘Place your wager’ box, maybe a text box and/or other numeric entry interface which may allow aplayer to designate a number of points. The wagering control 2450 b, the‘Bet’ button, may be used to confirm a wager. While a simple set ofwagering controls is illustrated in the example in FIG. 22, more complextypes of wagering controls might be provided for various types ofwagers, such as side-bets on the success or failure of other players,matching with other player's opinions, etc. within the scope of thedisclosure herein. Likewise, the GUI 2400 might allow a player notparticipating in a game to make wagers on various outcomes.

The mode control 2455 may be used to change the mode of the game beingplayed. For example, a player might elect to play against an expertopinion, a consensus opinion of various groups, a consensus of theplayers identified in the GUI 2400, a majority of voters who have votedregarding an item(s), any other type of reference group, such as only‘Novice’ players, or players from a geographic region, or players whofit a profile of any sort. The mode control 2455 may be implemented as adrop-down list, or other interface which may be used to designate a typeof game.

As illustrated in FIG. 23, an exemplary GUI is provided for selection ofa multi player game associated with a category, topic, word, phrase, orany combination thereof. The GUI 2500 includes availability selectioncontrol 2105, logoff control 2110, game information indicator 2115,activity selection tabs 2120, search button 2325, search box 2326, gamelocation tabs 2330, advertisement window 2135, game informationindicators 2540, recent game display 2345, recent game selectioncontrols 2347, scoreboard indicator 2150, action buttons 2555, gameselection controls 2560, game information window 2565, and navigationcontrols 2570. The GUI 2500 may be used to select a game(s) which may beplayed as a multi-player activity. Games may be selected using filteringcriteria. For example, games may be selected by entering and/orselecting a topic, keyword, category, etc or based on a skill level, orany other information which may be associated with a game. For example,games might be associated with an affiliate group, a profile, etc. Aprofile might include information such as a time interval, geographiclocation, personality characteristics, or any other item(s). Selectionsavailable may be based on information associated with a person(s). Forexample, some games may be available only to persons associated with anitem, or with a set of skills, etc.

The function of the availability selection control 2105, logoff control2110 game information indicator 2115, activity selection tabs 2120,scoreboard indicator 2150, the advertisement window 2135 and thescoreboard indicator 2150 were previously explained with respect to FIG.19. The function of the search button 2325, search box 2326, gamelocation tabs 2330, recent game display 2345, and recent game selectioncontrols 2347 were previously explained with respect to FIG. 21.

The content of the game information window 2565 may be based on theresults of a search performed responsive to a search term entered in thesearch box 2326 and executed using the search button 2325. Using theexample illustrated in FIG. 23, the search term ‘Indianapolis’ has beenentered in the search box 2326. The game information indicators 2560displayed in the game information window 2565 may indicate that the term‘Indianapolis’ is associated with games indicated in the gameinformation indicators.

The game information indicators 2540 may present information regardingtypes of information associated with a game(s). The ‘Game Topic’ gameinformation indicator 2540 a may indicate a category. A left-click onthe ‘Game Topic’ game information indicator 2540 a may sort games in thegame information window 2565 by categories associated with the game. Aright-click on the ‘Game Topic’ game information indicator 2540 a maypresent a drop-down list of categories. The ‘Skill Level’ gameinformation indicator 2540 b may indicate a skill. A left-click on the‘Skill Level’ game information indicator 2540 b may sort games in thegame information window 2565 by skill level associated with the game. Aright-click on the ‘Skill Level’ game information indicator 2540 b maypresent a drop-down list of skill levels. The ‘Players’ game informationindicator 2540 c may indicate a number of participants in a game. Aleft-click on the ‘Players’ game information indicator 2540 c may sortgames in the game information window 2365 by number of participantsassociated with the game. A right-click on the ‘Players’ gameinformation indicator 2540 c may present a drop-down list. Any number ofgame information indicators 2540 may be provided in the GUI 2300, and itmay be possible to add and/or remove game information indicators 2540.For example a ‘type’ indicator (comparative, good/bad, comparison, etc.)might be provided as a game information indicator 2540.

The action buttons 2555 may be used to take actions regardinginformation indicated in the GUI 2300. The ‘Go to Game’ action button2555 a may be used to activate a game GUI such as the GUI 2400illustrated in FIG. 22 associated with a selected game selection control2560. For example if the game selection indicator 2560 b is selected,activation of the ‘Go to Game’ action button 2555 a might present a GUIsuch as the GUI 2400 (FIG. 22) associated with the category‘Sports>Baseball’ at the ‘Expert’ level of skill. The action button 2555b may be used to view information of players associated with a gameselection indicator 2560. For example, if the game selection indicator2560 c is selected, activation of the action button 2555 b may cause alist of players associated with the game indicated in the game selectionindicator 2560 c to be displayed. Any number of action buttons 2555 maybe included in the GUI 2300.

The game selection controls 2560 may be used to select a game associatedwith a game selection control 2560. For example, the game selectioncontrol 2560 a may be used to select the ‘Novice’ level game associatedwith ‘Sports>Football’, a ‘left’ click might select the game selectionindicator 2560 a and a ‘double’ click might cause a GUI such as the GUI2400 (FIG. 21) associated with the ‘Sports>Football’ category at the‘Expert’ skill level to be presented. Likewise, the game selectionindicator 2560 c may be used to select the ‘Novice’ level gameassociated with the category ‘Entertainment>Dining>Indianapolis’, aleft-click might select the game selection indicator 2560 c and a doubleclick might cause a GUI such as the GUI 2400 (FIG. 24) associated withthe “Entertainment>Dining>Indianapolis’ category at the ‘Novice’ skilllevel to be presented. The navigation controls 2570 may be used tonavigate within the game information window 2565.

While the GUIs described herein have been described using specificexamples of interface elements, no limitation is implied thereby otherinterface elements which may accomplish the functionalities which arewell known in the art may be used without departing from the scope andspirit of the embodiments herein. Any or all of the elements of the GUIsdescribed herein may be suppressed as needed to operate the embodiments.

Using the GUIs described herein, a voting and review activities may beselected and presented as a form of entertainment, which may increaseparticipation and may improve utilization of resources associated with asearch system. Advertisements may be presented to players in the gamesbased on contextual information such as a topic, category, profile, orother information associated with reference items and/or review itemspresented as part of the gaming experience.

While specific examples of games which might be provided to reviewershave been presented, no limitation is implied thereby. Additional gameswhich might involve other forms of cooperation and/or competition mightbe implemented. A gaming activity may be an uncompensated activity, ormay be a compensated activity in which ‘points’ may be converted intosome form of compensation. The way in which points are converted tocompensation is not limited to a particular process.

A framework is provided whereby information may be persisted in thedatabase 120 (FIG. 1) which is independent of the form of game and/orreview which is used to produce the information. For example, ratingand/or ranking of a review item(s) may be associated with informationwhich may be passed back and forth to software and/or a system whichimplements a game using a defined data structure which is independent ofthe game application. Likewise, a rating of a guide may be independentof the types of games played. For example, a guide might play multiplegames which affected rating information associated with the guide orrating information of a guide might be based on information obtainedwith a game(s) and rating information of another guide might be based oninformation obtained using the same game(s) or a different game(s).Using this process, a guide may have a rating which is not dependent onany particular activity, but is linked to information associated withthe guide which is persisted.

A review item such as a search result, or a resource, etc. is associatedwith persistent information which identifies the status of the reviewitem. For example, if an answer is identified as a ‘best’ answer to aquery, that information is persisted after a review session has beencompleted. Usage of a review item may be based on the persistentinformation associated with the review item. Multiple review sessionsmay be used to determine information associated with a review item.

A method and system for reviewing or vetting an item(s) associated witha reference item is described. A voter or reviewer may have a voteweight which is determined based on a voting history associated with thevoter. A group of one or more reference voters, who may have a highervote weight, is identified. A rating function is computed based on acomparison of votes associated with a voter to votes associated with areference voter(s) and/or other voters. A group of voters may bedetermined based on the value of a rating function associated with thevoters. A vote weight is assigned to a voter based on the value of arating function associated with the voter. A vote weight and/or a ratingfunction may be associated with an item(s) associated with a voter, suchas a keyword(s), a category(ies), a profile(s), a tag(s), a topic(s),etc. which may be used to select a voter(s).

Voting by guides may be used to determine the ranking of review itemssuch as an answer, a search result, a search resource, a web page, anadvertisement, a category, a keyword, a query, etc. which may beassociated with a reference item such as a search request, a question, acategory, a web page, a profile, etc.

A method and system are described for comparative voting. A voter may bepresented with a reference item such as a query or search request,and/or other items and a choice of two or more review items which may bevoted on by the voter. A voter is allowed to compare the review itemssimultaneously and may be allowed to select a limited number of thereview items. Review items may be grouped in groups of two or more. Areview item which is determined to be a superior result may be requiredto meet specific criteria to be presented to a user and/or a guide(s)when a reference item is selected. A search result obtained by a humansearcher responsive to a search request may be required to be judgedsuperior to one or more other search results in order to be designatedas a superior result or ‘web gem’. A search result produced by a humansearcher responsive to a search request may be required to be ratedsuperior to each of 3 other search results produced responsive to thesearch request to be considered a Web Gem™.

A method and system are described for determining when human judgmentmay be required to evaluate a search result(s) or other item(s). Forexample, if a search request or search query has been submitted morethan a number of times, search results associated with the searchrequest may be reviewed by guides and/or users to determine a ranking ofsearch results associated with the search request. If a number ofanswers associated with a search request exceed a value, the judgment ofvoters may be used to determine a ranking of the answers. Usageindications of a reference item and/or a review item may be used todetermine whether human judgment should be used to review a referenceitem(s) associated with a review item(s). A review item which has beenpresented or accessed more than a number of times may be subjected toreview and/or voting by a guide(s) and/or a user(s).

Voters may be selected based on ratings associated with a referenceitem. For example, voters who are located in closer proximity to alocation associated with a search request may be preferentially selectedto vote regarding answers associated with the search request, or votersmay be selected based on a rating(s) associated with a category(ies), aprofile(s), a keyword(s), a topic(s), etc. which may be associated witha reference item.

A method for reviewing a search result associated with a search requestis described. If a search request has been submitted more than a numberof times, a first search result associated with a search request may bepresented to a reviewer to determine whether the search result isrelevant and if the reviewer determines that the first search result isnot relevant, the search request may be presented to a searcher or guidewho may obtain a second search result responsive to the search request,which may be compared to the first search result in order to determinewhether the first search result or the second search result is morerelevant.

Any of the voting methods and/or review methods described herein may beused separately or in conjunction with each other, and may improve therelevance of a search result, an advertisement, a search resource,and/or other items presented responsive to a search request.

The many features and advantages of the claimed invention are apparentfrom the detailed specification and thus, it is intended by the appendedclaims to cover all such features and advantages of the claimedinvention that fall within the true spirit and scope of the invention.Further, since numerous modifications and changes will readily occur tothose skilled in the art, it is not desired to limit the invention tothe exact construction and operation illustrated and described for thedisclosed embodiments, and accordingly all suitable modifications andequivalents may be resorted to, falling within the scope of the claimedinvention. It will further be understood that the phrase “at least oneof A, B and C” may be used herein as an alternative expression thatmeans “one or more of A, B and C.”

1. A method of determining relevance comprising: obtaining a reference;selecting, by a processor, an item based on the reference; presenting anindicator of the reference and an identifier of the item to a reviewer;receiving an opinion of the reviewer; assigning a weight to opinions ofthe reviewer based on a comparison of the opinion to a judgment of arepresentative; ranking items for a selection based on the weight andthe opinions.
 2. The method of claim 1 comprising: performing thecomparison by comparing the opinion to opinions received from reviewerspresented with the indicator and the reference; and assigning theweighting based on a number of the opinions in agreement with theopinion.
 3. The method of claim 1 comprising: presenting a question asthe indicator and a phrase and a Uniform Resource Locator as theidentifier.
 4. The method of claim 1 comprising: computing the weightbased on opinions of representatives associated with a categorydetermined by the reference.
 5. The method of claim 1 comprising:receiving the selection from a group of categories; and ranking therelevance of the items based on the weight and the opinions when thejudgment of the representative is received for the item.
 6. The methodof claim 1 comprising: receiving the selection from a group ofcategories; and ranking the relevance of the items based on the weightand the opinions when a guide has provided an evaluation of the item andthe evaluation of the guide has been affirmed by a comparison ofevaluations by the guide to judgments of the representative.
 7. Themethod of claim 1 comprising: linking the reviewer to a keyword;selecting the item based on the keyword; choosing the representativebased on a ranking of representatives for the keyword.
 8. The method ofclaim 7 comprising: administering a test to a guide; comparing judgmentsof the representatives to choices of the guide; and ranking therepresentatives based on the comparison.
 9. The method of claim 1comprising: associating a location with the representative; andassociating the weighting with the location.
 10. The method of claim 1comprising: obtaining judgments of the representative regarding groups;including in the groups a query and a response; receiving assessments ofthe groups including the reference and the item by users; and performingthe comparison based on correlation between the judgments, theassessments and the opinion.
 11. The method of claim 1 comprising:presenting the indicator and the identifier to the reviewer based on aranking of the reviewer for the reference; and assigning the weight tothe opinions associated with the reference.
 12. The method of claim 1comprising: selecting the item based on a ranking of the item for thereference; and presenting the identifier of the item in an order basedon the ranking.
 13. The method of claim 1 comprising: determining anumber of references associated with the reviewer; ranking thereferences based on a number of judgments provided for the reference;and presenting the references to the reviewer in an order based on theranking.
 14. A system comprising: a reviewer system presenting anindicator of a reference and an identifier of an item and receiving anopinion; a server system, comprising memory and a processor, selectingthe item based on the reference, obtaining the reference, assigning aweight to opinions, comparing the opinion to judgments, determining theweight based on the comparing, and ranking items for a selection basedon the weight and the opinions; and a representative system providingthe judgment.
 15. A non-transient computer readable medium storingtherein a program for causing a computer to execute an operationincluding rating a result, comprising: obtaining a reference; presentingan indicator of the reference and an identifier of an item selectedbased on the reference to a user; receiving an opinion of the user;presenting the indicator and the identifier to a human assistant;receiving a choice of the human assistant; determining a weightassociated with the opinion of the user based on the choice of the humanassistant.
 16. The computer readable medium of claim 15, the operationfurther comprising: determining the weight based on a ranking of thehuman assistant for the reference.
 17. The computer readable medium ofclaim 15, the operation further comprising: determining the weight basedon a number of times choices of human assistants agree with the opinionwhen the human assistants are presented with the indicator and theidentifier.
 18. The computer readable medium of claim 15, the operationfurther comprising: obtaining the reference based on a number of humanassistants who have provided choices associated with a categoryassociated with the reference.
 19. The computer readable medium of claim15, the operation further comprising: obtaining the reference based on anumber of items associated with the reference.
 20. The computer readablemedium of claim 15, the operation further comprising: determining theweight based on a comparison of choices of the human assistant toevaluations provided by a human guide.